RAR streaming development..via SMB/NFS

[lundman]

Even though Dune is no longer a NMT, it is virtually the same hardware. I want to add unrar support to Dune, and the best way for me to do so that I can see right now, is the LD_PRELOAD my own library to replace libc calls open/read/close/stat/lseek/opendir/readdir/closedir.

Reason I mention this is that it will be the same exact code to run on PCH as well, same uclibc, busybox, kernel etc. Perhaps others have thought about this already? Other ideas or solutions?

(Also found the other forums to be really slow compared to this one!)

[lundman]

Anyway, proof of concept is done:

Code:

# export LD_PRELOAD=/opt/sybhttpd/localhost.drives/HARD_DISK/Apps/librarcy/librarcy.so.1.0.1
# ls -l
-rw-r--r--    1 root     root         9149 Feb 27 14:30 librarcy.so.1.0.1
# cat /tmp/library.log
Successful intersection: stat64:

Now I just need to add the actual code. Anyone know how libc "normalises" the paths before using them, in open/stat/opendir?

[lundman]

Anyway, proceeding happily here. I have over-loaded opendir/readdir, and inject the media files whenever I find a RAR. I am cheating to make the development faster. I will just add the file.avi from file.rar, I could hide the .rar files, but then you can't copy or delete them.

Next is to change stat64/lstat64 to return the values I find from readdir, then open/read.

The tedious one will be lseek, but should not be too bad even.

Code:

# ls -l
-rw-r--r--    1 root     root        12843 Feb 27 19:51 librarcy.so.1.0.1
-rw-r--r--    1 root     root         3254 Feb 27 14:59 test.rar

# LD_PRELOAD=/opt/sybhttpd/localhost.drives/HARD_DISK/Apps/librarcy/librarcy.so.1.0.1 ls -l
-rw-r--r--    1 root     root        12843 Feb 27 19:51 librarcy.so.1.0.1
-rw-r--r--    1 root     root         3254 Feb 27 14:59 test.rar
-rw-r--r--    1 root     root          200 Jan  1 00:00 test.avi

[lundman]

Making good progress here, although I feel I am about a day behind my schedule that I just made up.

My program does opendir/readdir64/lstat64/stat64 just fine, and expand all .rar files

Although, amusingly, if I include the calls to read:

Code:

read: 513248-513252
read: 513252-539763
lseek: cur pos 539763 != 524288 wanted pos (whence 0: 00524288)
lseek: cur pos 524288 != 540672 wanted pos (whence 0: 00540672)
lseek: cur pos 540672 != 557056 wanted pos (whence 0: 00557056)
lseek: cur pos 557056 != 569500 wanted pos (whence 0: 00569500)
read: 569500-570524
lseek: cur pos 570524 != 557056 wanted pos (whence 0: 00557056)
lseek: cur pos 557056 != 573440 wanted pos (whence 0: 00573440)
lseek: cur pos 573440 != 589824 wanted pos (whence 0: 00589824)
lseek: cur pos 589824 != 599344 wanted pos (whence 0: 00599344)
read: 599344-600368
lseek: cur pos 600368 != 539778 wanted pos (whence 1: -0060590)
read: 539778-539782
read: 539782-569500
lseek: cur pos 569500 != 569515 wanted pos (whence 1: 00000015)
read: 569515-569519
read: 569519-599344
lseek: cur pos 599344 != 589824 wanted pos (whence 0: 00589824)
lseek: cur pos 589824 != 606208 wanted pos (whence 0: 00606208)
lseek: cur pos 606208 != 622592 wanted pos (whence 0: 00622592)
lseek: cur pos 622592 != 631960 wanted pos (whence 0: 00631960)
read: 631960-632984

I don't know at all whats going on there. Whats with the backward seeks? The multiples i can ignore, but going back is a respawn. The largest negative seek I found is -0114918. This means a buffer of 112KB at least, probably 128KB. This will take some extra time.

[cheaper_popcorn]

Quote:I don't know at all whats going on there.

Be that as it may, I still think your chances of figuring it out are greater than mine.

[chris57]

Thread moderated, edited and moved to Apps & Technical for better targeting.

[lundman]

Anyway, the basic code is done. If it is "happy-hour for lseek" I don't know how it will run (is the lseek usage based on .avi format?) but for the rest it should be enough to be functional.

One day I could extend it to work with sh in general, but currently you can't copy the files from archives out. (cat and cp uses fopen instead of open).

Test code:

Code:

fd = open(argv[1], O_RDONLY);
        if (fd >= 0 ) {
            while((red = read(fd, buffer, sizeof(buffer))) > 0)
                write(1, buffer, red);
            lseek(fd, 1, SEEK_SET);
            while((red = read(fd, buffer, sizeof(buffer))) > 0)
                write(1, buffer, red);
            close(fd);

Test execution output:

Code:

$ LD_PRELOAD=./librarcy.so.1.0.1 ./tester minicom.log
20081226 16:17:51 Hangup (0:00:00)
0081226 16:17:51 Hangup (0:00:00)

Debug log output:

Code:

looked up 'open64': 0x2ab82070
[findfile] looking for 'minicom.log' ...
[findfile] archive 'test.rar' has file 'minicom.log'.
[open64] open on archive entry for 'test.rar/minicom.log'.
[fdmap] linked 'minicom.log'
[spawn] launching '/tmp/unrar p -inul -c- -p- -y -cfg- -sk0 -- "test.rar" "minicom.log"' ...
[spawn]: fd 3 spawned.
[read] fd 3 read 70 bytes (0->70)
looked up 'lseek64': 0x2ab81500
[lseek] fd 3 seek whence 0 amount 1
[read] fd 3 lseek call detected: respawning
[despawn] stopping previous unrar
[spawn] launching '/tmp/unrar p -inul -c- -p- -y -cfg- -sk1 -- "test.rar" "minicom.log"' ...
[spawn]: fd 3 spawned.
[read] fd 3 read 69 bytes (1->70)
[close] finished with fdmap 3
[despawn] stopping previous unrar
[fdmap] releasing 'minicom.log'

[mercenary]

lundman, will this work allows RARs to be playable without LLINK? and would it make them appear and be manipulated just as AVIs? would help alot for all the jukeboxes out there

[lundman]

Hmm, well, it starts playing, for about 1 second (the first buffer of 65k) then it thinks its finished.

Code:

[read] fd 6 lseek call detected: respawning
[spawn] 1 fdmap nodes.
[despawn] stopping previous unrar
[spawn] launching '/tmp/unrar p -inul -c- -p- -y -cfg- -sk0 -- "/The.Simpsons.S20E11.HDTV/the.simpsons.2011.rar" "the.simpsons.s20e11.hdtv.xvid.avi"' ...
[spawn]: fd 6 spawned.
[read] fd 6 read 65536 bytes ( 0->65536)
File ready 1 times, waiting for EOS
   wait 2 events [0x3 = 0x00020000] [0x101a = 0x00000008]
   get 0x101a = 0x00000008
inband_event 0x101a with tag=20010
WaitForEOS 0x101a OK
inband_event 0x101a without tag - hwlib already process it
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   wait 1 events [0x3 = 0x00020000]
   get 0x3 = 0x00020000
display_inband_event with tag=20010
WaitForEOS display OK
inband_event display without tag - hwlib already process it
EOS_InbandCmd after 926513 us

Not entirely sure how it keeps playing, perhaps it hands it off somewhere, which doesn't call read()... I need to go deeper..

[quassum]

Would be amazing if you would be able to pull this one off lundman! Curiously following your steps over here, sorry I can't help you with anything but if you need testers/feedback please let me know.

[lundman]

That was actually a bug in my lseek handling, for some reason RUA loves to call lseek multiple times without read, and not just to do the weak "filesize check".

Anyway, playing just the .avi file and playing the .avi in rar, and comparing the strace, I have it so that they run as identical as expected. But for some reason when playing from RARs, it does the initial buffer load, and starts playing...

However, then the plain AVI will lseek(CUR,0) and read the next buffer fill, the RAR version just says:
"Reach end of file, waiting for EOS."

... which comes from play_avi_push.c, the "fill_buffer:" goto. However, there are so many ways it can land there I am not entirely sure what goes wrong. No unusual calls besides RMFileSeek and RMFileRead in that function, which are handled.

Anyone know if I can turn on RMDBGLOG() so I can see what it says without recompiling? Or is it just a compile-time define?

[lundman]

Code:

read(6,0x2bc7b000,524288)
[read] fd 6 read 65536 bytes ( 0->65536)

They ask for 524288 and _just assume_ that read() will read it, or EOF! I will refrain from making comments...

But yes, read of a pipe shouldn't stop at 65536 (but it does) and Sigma shouldn't assume read() will always return the full buffer (even when blocking). But anyway, I now re-call read until the buffer is full, and the video plays completely.

Apart from bells & whistles, the work is done.

[quassum]

Wow amazing, also close to a beta release, can't wait to test this!

[werner]

nice work lundman

[lundman]

I have watched a couple of things on the Dune without any problems, seeking works too. I will try on the PCH as well just to see how it does.

but I did notice that librmavipush has:

Code:

/* Open with caching if it is HTTP, because we seek a lot during
playback */                err = set_http_options(&stream_options, RM_HTTP_OPEN_CACHED, NULL);

So maybe relatively little change would be needed to open with caching. Who knows.