xorriso: extracting files to stdout

[G.raud]

Is it possible with xorriso to extract one file (or even cating several
files) from an iso image to stdout (or to a given filename that would
not be unlinked first like a fifo or /dev/fd/1 or to any file descriptor
or maybe to a pipe to a supplied command to exec)?

If not would it be something reasonable (i.e. not bloat) for xorriso to
be able to do?

[Thomas Schmitt]

Hi,

Post by G.raud
Is it possible with xorriso to extract one file (or even cating several
files) from an iso image to stdout (or to a given filename that would
not be unlinked first like a fifo or /dev/fd/1 or to any file descriptor
or maybe to a pipe to a supplied command to exec)?

Currently not. But it would be doable.

The unconditional attempt to delete the extration target file
before writing the extracted data, is a security measure against
symbolic link traps. (As malicious user create a link to a file
which does not offer permission. Then lure an authorized user
into writing data to the link.)

One could lift that precaution for named pipes, block devices,
and character devices.
/dev/fd/1 is supposed to be a symbolic link pointing to some
/dev/pts/NNN. So the ban would also have to be lifted for
symbolic links with appropriate target file types.

So you see any use case where a regular data file should be
overwritten rather than being replaced ?
(Other than the existing xorriso command -paste_in, which
composes a disk file from several data files in the ISO.)

-----------------------------------------------------------------

The removal of the existing file is performed in
xorriso/read_run.c : Xorriso_restore_overwrite()
by
ret= Xorriso_rmx(xorriso, (off_t) 0, path, 8 | (flag & 64));

Then there is a safety precaution against writing to existing files in
xorriso/read_run.c : Xorriso_tree_restore_node()
by
if(disk_offset==0 || !(flag&2))
open_flags|= O_EXCL;

If these two are disabled, then

-extract /my/file /dev/fd/1

writes the content of /my/file to stdout.

-----------------------------------------------------------------

I will put this onto my todo list.

There remains the problem, that xorriso might also write text messages
to stdout, which would be hard to distinguish from file content.
A similar situation is given with ISO image output to stdout. Probably
i can derive safety precautions from that occasion.

How urgent is this feature to you ?


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

I want to pass a fixed list of filenames to list. The best way seems to
be to use -wholename and hope that filenames do not contain special
xorriso -indev image.iso -return_with WARNING 32 -report_about WARNING \
-find / -wholename "$name1" -o -wholename "$name2" -exec lsdl
Is there a way to avoid the interpretation of these patterns by xorriso
or another way to list file with a fixed-string filename?

Currently not. (Again :))

Do i get it right that you want to get a non-zero exit value from
the program run if none of the files in the list exists in the ISO ?

Maybe you would be better off with a simple -ls command while
wildcard expansion is disabled by -iso_rr_pattern "off":

$ xorriso -indev /some/non-ISO \
-return_with WARNING 32 -report_about WARNING \
-iso_rr_pattern off \
-ls "/such*.c" --
GNU xorriso 1.3.7 : RockRidge filesystem manipulator, libburnia project.

xorriso : WARNING : Not found in ISO image: '/such*.c'

although a file /such.c does exist in the ISO image. It would be
listed without warning if -iso_rr_pattern was set to "on" or to "ls".

The -return_with WARNING is there to fail if no file could be found
(which is the case if image.iso is not a valid iso).

You already get a warning when loading a non-ISO-image.

$ xorriso -indev /some/non-ISO \
-return_with WARNING 32 -report_about WARNING
GNU xorriso 1.3.7 : RockRidge filesystem manipulator, libburnia project.

libisoburn: WARNING : No ISO 9660 image at LBA 0. Creating blank image.
$ echo $?
32

The commands -report_about and -return_with are already in effect
when the interpretation of -indev happens. That's because more than
one such setting might be desired. By this rule they do not have to
fight for being first command in the list.
(Normally commands get into effect only after all previous commands
in the argument list have been executed.)

If you really want to tolerate a WARNING with -indev and later take
WARNING as cause for a non-zero exit value, then you first have to
set the event thresholds to a higher severity for the duration
of the -indev command:

xorriso -return_with FAILURE 32 -report_about WARNING \
-indev /some/non-ISO \
-return_with WARNING 32 \
-find ...


If you just want to know whether a data file is an ISO image
or not, you could also use the shell command "file"

$ file test.iso
test.iso: ISO 9660 CD-ROM filesystem data '...Volume.Id...'

$ file not_an_iso
not_an_iso: ASCII text

-------------------------------------------------------------------

Maybe adding a -find_pattern option or making -iso_rr_pattern enable/disable
the use of patterns by -find would be a good idea?

For compatibility reasons, it would inot be appropriate to extend
the effect of -iso_rr_pattern and -disk_pattern to -find tests.

But one could introduce pseudo tests -iso_rr_pattern and -disk_pattern
which always return true and control pattern interpretation. Like:

-find / -iso_rr_pattern off ...more.tests.and.one.action...

The system find has a similar issue, but there is a way to find fixed
filenames by using the option -exec

In xorriso's -find, -exec is not a test and can only appear once.

Would you consider adding a -fexec to your -find to use a system command
to test a filename (and maybe more than the filename)?

That would be applicable to disk files, but not to files in the
ISO image.
It would be quite some effort to enable multiple -exec with arbitrary
xorriso commands and to attribute exit values to those commands.
This feature of POSIX "find" depends on the concept of a shell resp.
of POSIX program execution. xorriso would have to mimic this.


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

Post by Thomas Schmitt
So you see any use case where a regular data file should be
overwritten rather than being replaced ?

I meant "Do" rather than "So":
Do you see any use case where a regular data file should be
overwritten rather than being replaced ?


Musings about -extract and unlinking its target files:

In any case the default of xorriso will stay "unlink any file".
Now i ponder about the non-default options.

The permission to follow symbolic links would have to be separated
from the permission to keep a target file of suitable type linked.
(If my above corrected question yields "yes" then there would be the
need for a third permission: "do not unlink regular files".)

Further the extraction source might be no regular data file. In this
case it will not be possible to write its content into existing
file objects on hard disk.
If in "do not unlink" mode, shall this lead to refusal or shall
the target object be unlinked and replaced by a copy of the source ?


Have a nice day :)

Thomas

[G.raud]

Date: Wed, 16 Apr 2014 09:20:04 +0200
Subject: Re: xorriso: extracting files to stdout

Post by G.raud
Is it possible with xorriso to extract one file (or even cating
several
files) from an iso image to stdout (or to a given filename that would
not be unlinked first like a fifo or /dev/fd/1 or to any file
descriptor
or maybe to a pipe to a supplied command to exec)?

Currently not. But it would be doable.

Good. I was wondering whether you had resorted to seek into the
extracted files to write the data in several passes because of potential
complexities.

One could lift that precaution for named pipes, block devices,
and character devices.
/dev/fd/1 is supposed to be a symbolic link pointing to some
/dev/pts/NNN. So the ban would also have to be lifted for
symbolic links with appropriate target file types.

/dev/fd/<n> may probably be a non symlink, and it points to /dev/pts/
when the corresponding FD is a pseudo-terminal (and the system uses UNIX
98 pseudo-terminals). On Linux, a pipe is a symbolic link to nowhere
and it is unlikely that all UNICES would output something similar:

$ ls -l /dev/fd/ |( cat; ls -l /dev/fd/ )
total 0
lrwx------ 1 graud graud 64 Apr 17 0707:28 0 -> /dev/pts/15
l-wx------ 1 graud graud 64 Apr 17 0707:28 1 -> pipe:[17009189]
lrwx------ 1 graud graud 64 Apr 17 0707:28 2 -> /dev/pts/15
lr-x------ 1 graud graud 64 Apr 17 0707:28 3 -> /proc/11471/fd/
total 0
lr-x------ 1 graud graud 64 Apr 17 0707:28 0 -> pipe:[17009189]
lrwx------ 1 graud graud 64 Apr 17 0707:28 1 -> /dev/pts/15
lrwx------ 1 graud graud 64 Apr 17 0707:28 2 -> /dev/pts/15
lr-x------ 1 graud graud 64 Apr 17 0707:28 3 -> /proc/11474/fd/

I don't think you should try to guess what a user might want to do, even
if it is legitimate to prevent misuse, because that is often what breaks
the versatility of (an interface of) a piece of software. For example
in our case, the chasing of symlinks inside /dev/fd/ (which is
/proc/<pid>/fd/ in fact) does not work as in a regular filesystem:

$ cat </dev/null |
( set -ex; /usr/bin/test -p /dev/fd/0;
/usr/bin/test -p /dev/fd/$(readlink /dev/fd/0); echo true )
+ /usr/bin/test -p /dev/fd/0
++ readlink /dev/fd/0
+ /usr/bin/test -p '/dev/fd/pipe:[17061066]'
# on a regular storage fs, this should have succeeded or failed
# early because the link does not contain any path component
$ cat </dev/null |chase /dev/fd/0
chase: /proc/31147/fd/pipe:[17103574]: No such file or directory
$ cat </dev/null |/usr/bin/test -e /dev/fd/0 && echo true
true

So you see any use case where a regular data file should be
overwritten rather than being replaced ?
(Other than the existing xorriso command -paste_in, which
composes a disk file from several data files in the ISO.)

1. Files that are deliberately not regular: fifo (named pipe), pipe, pty,
/dev/null (could be used to test the readability of a file or to
simply discard it), /dev/full (to test the error detection capacity
of xorriso?), symbolic link to a regular file or to a non existing
file (some users might like to use those to dispatch the writing
across filesystems, to redirect unneeded extracted data to /dev/null,
to concat several files to the same regular file, ...), block device
(to restore a backup of a USB key for example), special device
corresponding to a non standard driver (assume that the user knows
what he is doing).

2. Files that cannot be removed/recreated/chased (1. also matches this
category):
- filesystems under the control of special drivers, like a FUSE
filesystem to access a network resource or the previously mentionned
/proc filesystem;
- if the directory permissions does not allow the user to remove/create
files but a specific file was created for his use with the
appropriate permissions to write to it.

3. Concatenating data streams by appending to a file (example:
/VIDEO_TS/VTS_01_*.VOB on a DVD, similar to -paste_in I suppose).

4. Avoiding race conditions: an independent program might test that a
file does not exist just after xorriso removed it and use the name;
letting xorriso overwrite a file would put the responsibility of
avoiding such race conditions in the hand of the user (if he so
desires).

5. Anything else that I could not guess.

There remains the problem, that xorriso might also write text messages
to stdout, which would be hard to distinguish from file content.

This is easy to avoid if one can write to any FD and not just to stdout.
For example to wrap xorriso so that stdout is sent to stderr (here the
terminal) and FD 4 is redirected to stdout (here the pipe):

$ sh -c 'exec 4>&1 1>&2 "$0" "$@"' xorriso other args -cat "$name" /dev/fd/4 |
cmd of my own that no one else could have thought of |
to the network without touching the filesystem

A similar situation is given with ISO image output to stdout. Probably
i can derive safety precautions from that occasion.

I do not understand what you mean by those sentences.

How urgent is this feature to you ?

Not urgent. You should take the time to decide which way is the best to
allow to do use the full potential of the UNIX file abstraction.
Particularly if you care for backward compatibility as you would not
revise your choice.

I would say the simplest way is to add a new command -cat that would be
similar -cpx but that would disable any smartness and simply append
to the given file and not try to set any metadata (thus skip special
files or fail on them). Either let it take 2 args: the iso filename and
the output or one, the iso filename, and add a global option, for
example -catfile, to redirect the output of the command -cat to
something else than stdout. -cpx would be used to try to recreate a
file system object and -cat to retrieve the file data.

I would also suggest allowing to create pipes directly in xorriso for the
convenience of users, particularly if stdout is not usable directly as
input to a pipe. For example with the following syntax '|shell command
string'. And maybe accept FD numbers with '|<n>' for the unlikely case
that the system does not have /dev/fd/<n>.

The permission to follow symbolic links would have to be separated
from the permission to keep a target file of suitable type linked.
(If my above corrected question yields "yes" then there would be the
need for a third permission: "do not unlink regular files".)

I think having options to change the behaviour of -cpx, -extract etc. so
that they accomodate what they find on the filesystem instead of adding
-cat_r etc. would be another sensible choice. -cat would still be
useful be useful for concatenating some files in the order specified on
the command line.

According to my previous experiments with the chasing of pipes (or
probably anything that is not on a special filesystem like a FUSE
filesystem), you should first get the file type then try to follow the
link if you really think there is a need to do so yourself, but
should'nt open() be sufficient (when the file pointed to does exist)?

In the case that symlinks are followed (or that they are not treated
especially), why still unlink the destination (except possibly for non
regular files on the iso)? Isn't choosing whether to append or truncate
and whether to set FS attributes sufficient?

Further the extraction source might be no regular data file. In this
case it will not be possible to write its content into existing
file objects on hard disk.
If in "do not unlink" mode, shall this lead to refusal or shall
the target object be unlinked and replaced by a copy of the source ?

The less surprising would be to report something. Another option would
be to always skip special files when the user requested only file data
without metadata. This choice should match the spirit of xorriso
however, and I do not know it very well. In any case the user should
have options to detect any possible discrepancy between the iso and what
is extracted if that is not the default.

Regards

[Thomas Schmitt]

Hi,

Post by G.raud
l-wx------ 1 graud graud 64 Apr 17 0707:28 1 -> pipe:[17009189]

Strange link target. What does this mean ?

Post by G.raud
I don't think you should try to guess what a user might want to do, even
if it is legitimate to prevent misuse,

I have to stick to the principle of least surprise.
It is well standard among backup systems to remove the target by
unlink() before writing data to its new incarnation.
This does not only avoid symlink traps, but also restores the
file system tree as it was, with no influence by the intermediate
state that is undesirable enough to call for the backup.

Post by G.raud
the chasing of symlinks [...] does not work

I will not have to lookup the link target myself.
But i have to decide whether the symlink shall survive
and its target shall be altered instead.

Post by G.raud

Post by Thomas Schmitt
Do you see any use case where a regular data file should be
overwritten rather than being replaced ?

if the directory permissions does not allow the user to remove/create
files but a specific file was created for his use with the
appropriate permissions to write to it.

That's a valid case, yes.

Post by G.raud
/VIDEO_TS/VTS_01_*.VOB on a DVD, similar to -paste_in I suppose).

If there is output to stdout or named pipes, then concatenation
will be available.
(-paste_in relies on random write access to the target file.)
Interesting aspect. Looks like a valid case.
Is there any function for atomic unlink-and-create ?

Post by G.raud
5. Anything else that I could not guess.

That's not a valid case. :))

Post by G.raud

Post by Thomas Schmitt
There remains the problem, that xorriso might also write text messages
to stdout, which would be hard to distinguish from file content.
A similar situation is given with ISO image output to stdout. Probably
i can derive safety precautions from that occasion.

I do not understand what you mean by those sentences.

I already have a situation where messages and binary output compete
for stdout:
xorriso -outdev - ...commands...
In this case, i direct all stdout messages to stderr.

I will investigate whether this is appropriate for file extraction
to stdout, too.

Post by G.raud
I would say the simplest way is to add a new command -cat

This has its appeal, yes. Something like

-cat file1 ... fileN target --

where target is allowed to be "-" for stdout.
This one could follow links, maintain existing file objects with
their attributes, refuse on target types which cannot take data,

The only problem is that it will not truncate data files to 0 size
before writing to them. (A hypothetical "-dd" could.)

So modifying settings for the existing extraction commands
will still be desirable.

Post by G.raud
I would also suggest allowing to create pipes directly in xorriso for the
convenience of users,
[...] '|shell command string'

I am not very comfortable with this, currently.
Starting shell commands is extending the potential damage range
of xorriso. Well, there are the external filters already.
I soothed my scruples by adding configure options to ban them
and by disallowing them to setuid users by default.
(--disable-external-filters, --enable-external-filters-setuid)

... pondering ...

Post by G.raud
I think having options to change the behaviour of -cpx, -extract etc. so
that they accomodate what they find on the filesystem instead of adding
-cat_r etc. would be another sensible choice.

That would be

- Follow symbolic links to replace or overwrite its target
rather than replacing the link itself.

- Do not unlink special files but rather overwrite their content
while keeping all their POSIX file attributes.
(What about ACL and getfattr attributes ?)

- Do not unlink regular files but handle them like the special
file in the previous option.

In case of no-unlink:
If the existing target file cannot take content because of file
type or permissions, then this is a failure event. The target
will not be replaced.
Special files from the ISO, which do not deliver content, will
not replace target files on disk but rather cause a failure event.

Post by G.raud
In the case that symlinks are followed (or that they are not treated
especially), why still unlink the destination (except possibly for non
regular files on the iso)?

Because one may want its attributes to be replaced, or because
one wants new ctime, or becaue of reasons which we both don't
have in mind. :))
Actually its two different things, which i just want to keep
separate.

Post by G.raud
This choice [of reaction on unfulfillable wextraction ishes]
should match the spirit of xorriso however, and I do not know
it very well.

The intended spirit is to be of help. :)
Too much eagerness might do more harm than good, though.

I will ponder over the easter holidays.

(Do you want a tarball for the new -find options to play with ?)


Have a nice day :)

Thomas

[Thomas Schmitt]

Hi,

i have now uploaded

http://www.gnu.org/software/xorriso/xorriso-1.3.7.tar.gz

with -find pseudo tests -use_pattern , -or_use_pattern,

-use_pattern "on"|"off" : This pseudo test controls the
interpretation of wildcards with tests -name, -wholename, and
-disk_name. Default is "on". If interpretation is disabled by
"off", then the parameters of -name, -wholename, and -disk_name
have to match literally rather than as search pattern. This
test itself does always match.

-or_use_pattern "on"|"off" : Like -use_pattern, but
automatically appending the test by -or rather than by -and.
Further the test itself does never match. So a subsequent test
-or will cause its other operand to be performed.

and with new command -concat (which needs to be enabled by -osirrox on):

-concat mode [target | lim prog [args [...]] lim] iso_rr_path [***]
Copy the data content of one or more data files of the ISO image
into a disk file object, into a file descriptor, or start a
program and copy the data into its standard input. The latter
is subject to the security restrictions for external filters.

Modes overwrite and append write into the target which is given
by the second parameter. This may be the path to a disk file
object, or "-" which means standard output, or a text of the
form /dev/fd/number, where number is an open file descriptor
(e.g. standard error is /dev/fd/2). An existing target file is
not removed before writing begins. If it is not able to take
content data, then this command fails. Mode overwrite truncates
regular data files to 0 size before writing into them. Example:
-concat append /home/me/accumulated_text /my/iso/text --

Mode pipe expects as second parameter a delimiter word which
shall mark the end of the program argument list. The third
argument is the disk_path to the program. It must contain at
least one '/'. $PATH is not applied. Further parameters up to
the announced delimiter word are used as arguments with the
program start. Example:
-iso_rr_pattern on \
-concat pipe + /usr/bin/wc + "/my/iso/files*" --

The further parameters in all modes are the iso_rr_paths of data
files. Their content gets concatenated in the copy.

The mentioned security restrictions are this command:

-close_filter_list
Irrevocably ban commands -concat "pipe", -external_filter, and
-unregister_filter, but not -set_filter. Use this to prevent
external filtering in general or when all intended filters are
registered and -concat mode "pipe" shall be disallowed.
External filters may also be banned totally at compile time of
xorriso. By default they are banned if xorriso runs under
setuid permission.

and these ./configure options (documented in README):

xorriso allows to use external processes as file content filters. This is
a potential security risk which may be avoided by ./configure option
--disable-external-filters

By default the filter feature is disabled if effective user id and real
user id differ. This ban can be lifted by
--enable-external-filters-setuid

Test reports are welcome.


Have a nice day :)

Thomas