• All submissions to this site are governed by Second Life Project Contribution Agreement. By submitting patches and other information using this site, you acknowledge that you have read, understood, and agreed to those terms.
Issue Details (XML | Word | Printable)

Key: VWR-5708
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Unassigned
Reporter: Xaden Kjeller
Votes: 18
Watchers: 9
Operations

If you were logged in you would be able to see more operations.
1. Second Life Viewer - VWR

Incorrect handling of /proc/cpuinfo in libvivoxsdk.so.

Created: 18/Mar/08 06:58 AM   Updated: 05/Feb/09 02:54 PM
Component/s: Voice
Affects Version/s: 1.21
Fix Version/s: 1.22 Release Candidate

File Attachments: 1. File cpuinfo (1 kB)
2. File fix_cpuinfo.sh (0.2 kB)
3. Text File gdb_backtrace.log (2 kB)
4. File override.c (2 kB)
5. File repro_cpuinfo.sh (0.2 kB)
6. Text File SecondLife.log (43 kB)
7. Text File siplog.txt (0.1 kB)
8. Text File strace.log (194 kB)

Environment:
GNU/Linux 2.6.23-gentoo-r8
Also observed on Arch Linux and Ubuntu 8.10 Beta
Issue Links:
Duplicate
Relates
 


 Description  « Hide
Overview:

In this bug report I will be outlining an issue I have discovered pertaining to the handling of /proc/cpuinfo by libvivoxsdk.so.
This bug results in the termination of the Vivox daemon (SLVoice) when an std::out_of_range exception is thrown by std::string::substr. I will include steps to reproduce the problem as well as a temporary solution for interested users.

Background:

When using the 1.19.0* and 1.19.1* clients I found myself unable to utilize features relating to voice communication. I noticed that the client was executing SLVoice but that the resulting process soon died. The output from the client on stdout and in the logs indicated that the client was reaching the stateConnecting internal state but that no further progress was made. I also witnessed output informing me that an std::out_of_range exception had been thrown. After checking the Vivox logs in ~/Library/Logs/Vivox/ I noticed the same output regarding an std::out_of_range exception in siplog.txt.

Investigation:

I decided to read the relevant sections of the Second Life client source code (llvoiceclient.cpp) to familiarize myself with its operation. I discovered that the client was requesting a "connector" object (LLVoiceClient::connectorCreate) from the daemon but would end up waiting for a response that never came.

Running SLVoice via strace (see attached) I confirmed that the daemon was successfully receiving requests from the client but aborted before a response could be sent. I also noticed that prior to its termination the last few system calls included a successful attempt to open and read from /proc/cpuinfo.

My next step was to generate a backtrace (see attached) of the stack in SLVoice using gdb. This revealed that the fatal call to std::string::substr originated in oscommon2::SplitNameValueFile called from oscommon2::MachineInfo::MachineInfo, both within libvivoxsdk.so. These names corresponded nicely with what I was seeing in the output of strace; with /proc/cpuinfo being regarded as "machine" information and being formatted with each line containing a "name" and "value" separated by a colon.

I soon came to the assumption that the problem lied with the handling of the /proc/cpuinfo format by libvivoxsdk.so in SplitNameValueFile. I decided the quickest way to test this hypothesis would be to create a copy of /proc/cpuinfo that was writable and to modify the string representation of the path to this file in libvivoxsdk.so (edited with vim in binary mode).

After a short period of experimentation I found that the exception was thrown if and only if a line terminated directly after a colon where a value was to be expected. If you look at the attached copy of my /proc/cpuinfo you can see that this occurs on the lines containing 'power management:\n'. Simply by removing these lines or modifying them to append a space to the colon I was able to successfully communicate with other residents via the Vivox daemon using voice communication.

Reproduction:

Reproducing this error requires that you run a GNU/Linux OS with the proc filesystem mounted and that the file /proc/cpuinfo contains at least one line in the form of 'name\t*:\n'.

If the latter is not the case then I propose that the simplest means to verify this is to copy the content of /proc/cpuinfo into a file with write permission, append to it a line in the above form (i.e. 'power management:\n') and modify the string representation of the path to cpuinfo within libvivoxsdk.so to point to the new file (either absolute or relative to the SLVoice executable).

The following is an example of a series commands that, when executed within the lib directory of the Second Life client's directory hierarchy, will perform the necessary operations to accomplish such a reproduction:

1. cat /proc/cpuinfo > cpuinfo
2. echo 'power management:' >> cpuinfo
3. cp libvivoxsdk.so libvivoxsdk.so.bak
4. sed -i 's/\/proc\/cpuinfo/lib\/cpuinfo\c@\c@/g' libvivoxsdk.so

I have wrapped these commands in a simple shell script which I have attached to this report: repro_cpuinfo.sh.

Temporary Solution:

For users who have identified that their system is affected by this issue I present the following series of commands (again assumed to be executed from within the client's lib dir) that work in much the same manner as the reproduction process above.

First, the data that comprises /proc/cpuinfo is copied (redirected) to a new, writable file; during this process a single space is appended to each colon which appears directly at the end of a line. Next, a copy of libvivoxsdk.so is created, its purpose is to serve as a backup. Finally, libvivoxsdk.so is overwritten to contain the modified path to cpuinfo.

1. sed 's/:$/: /g' /proc/cpuinfo > cpuinfo
2. cp libvivoxsdk.so libvivoxsdk.so.bak
3. sed -i 's/\/proc\/cpuinfo/lib\/cpuinfo\c@\c@/g' libvivoxsdk.so

I have wrapped these commands in a simple shell script which I have attached to this report: fix_cpuinfo.sh.

Conclusion:

It would appear that an incorrect assumption that a value will follow a colon in /proc/cpuinfo results in an out of bounds start position being passed to substr.

I hope that it is now apparent that the parsing of /proc/cpuinfo in libvivoxsdk.so needs to be improved. A more robust method of separating lines into name / value components is required so that it (the lib) may gracefully handle lines that do not conform to: 'name\t*: value\n'. The Second Life client successfully handles such cases (see LLCPUINFO::LLCPUINFO() in llsys.cpp) by simply discarding lines that do not match the desired format.

If this issue can be brought to the attention of the people at Vivox responsible for SLVoice/libvivoxsdk.so so that they can fully investigate and resolve this issue I would, as would the rest of the Second Life/Linux community, greatly appreciate it. I hope you find this information useful, please let me know if you require additional information.



 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
Saphirelda Jewell added a comment - 18/Mar/08 07:38 AM
Hi there! I'm from Vivox, and I have forwarded your info to Development. Thank you for all the detail and time researching this! We may contact you for more info if that is ok.

Saphirelda (aka Nancy)


Xaden Kjeller added a comment - 18/Mar/08 07:51 AM - edited
Thanks for the quick response. I just added a copy of my 'cat /proc/cpuinfo' output which I forgot to attach earlier. If I need to be contacted for further information that is not a problem.

HeXiLeD Zhangsun added a comment - 18/Mar/08 11:16 AM
I have the exact same problem and i had to apply the same fix:

Code:
#! /bin/sh
sed 's/:$/: /g' /proc/cpuinfo > cpuinfo
cp libvivoxsdk.so libvivoxsdk.so.bak
sed 's/\/proc\/cpuinfo/lib\/cpuinfo\c@\c@/g' libvivoxsdk.so.bak > libvivoxsdk.so

And it fixed the sound for me.
Gentoo Linux 2.6.23.9 - x86_64 Intel(R) Pentium(R) 4 Dual core - 64bit Install


Xaden Kjeller added a comment - 03/Apr/08 03:23 AM - edited
This issue is still present in the latest release (1.19.1.4). I'm leaving the affected version (after reading some advice from Torley Linden posted in another jira issue) solely at 1.19.0.5 as this is the release the issue originates from and was first observed.

Andros Baphomet added a comment - 02/Jun/08 01:19 PM - edited
Script fix_cpuinfo.sh confirmed to fix voice on 1.19.1.4. OS is Ubuntu 8.04 using pulseaudio. System details follow:

Computer
Processor : AMD Sempron(tm) 2200+
Memory : 1035MB (251MB used)
Operating System : Ubuntu 8.04
Display
Resolution : 1280x960 pixels
OpenGL Renderer : GeForce 6600 GT/AGP/SSE/3DNOW!
X11 Vendor : The X.Org Foundation
Multimedia
Audio Adapter : Audigy - Audigy 1 [SB0090]
Audio Adapter : USB-Audio - USB Device 0x46d [Logitech IM QuickConnect webcam]


Xaden Kjeller added a comment - 15/Jun/08 03:07 AM - edited
Setting the resolution as per http://jira.secondlife.com/browse/VWR-4589 now that a Linden has put their weight behind this issue. The ball remains in Vivox's court.

Note to affected users: You still need to workaround the issue (which persists through 1.19 and into the latest 1.20 Release Candidate, currently RC10) until Vivox addresses the issue in their library and it is included with one of the viewers. When that happens I will comment here with the details.


Tofu Linden added a comment - 16/Jun/08 08:40 AM
We have a fix from the vendor - it's creeping through QA now, no specific ETA.

Ochi Wolfe added a comment - 05/Oct/08 11:26 AM
Why is this issue marked resolved with "won't finish"? This issue is still persistent in the released 1.20 and the latest RC viewers. How many more months will QA take on this?

Also the workaround still works for me, but this can't be a permanent solution.


Jarv Linden added a comment - 03/Nov/08 02:33 PM
I just upgraded my macbook pro to Ubuntu 8.10 final and ran into this problem. I had previous sound-related problems (from a combination of Ubuntu 8.04 + the Macbook Pro's Intel HDA audio chip) that required swapping pulseaudio for esound. With 8.10, pulseaudio problems seem to be fixed, but I had to run this patch to fix voice support.

Thanks Xaden! Great job on the temporary fix.


Aleric Inglewood added a comment - 04/Nov/08 06:45 AM - edited
Not sure if this is to interest to anyone, since I read here that there is a fix from the vendor (why can't we just get the new libvivoxsdk.so?).
Anyway, I used this method: rename SLVoice to SLVoice-bin. Create a new file called SLVoice in the same place. This should be an
executable script with the contents:

#! /bin/sh
export LD_LIBRARY_PATH=/usr/local/lib32:/lib32:/usr/lib32
LD_PRELOAD=/usr/games/liboverride.so /usr/games/SLVoice-bin -p tcp -h -c -ll -1

Change /usr/games path as appropriate.
Note that you can also insert strace here, if you want.

The /usr/games/liboverride.so is compiled from override.c as follows:

gcc -m32 -g -D_GNU_SOURCE -shared -fPIC override.c -o liboverride.so -ldl

The -m32 is only needed if you're running 64bit, of course (the SLVoice-bin has
to be 32bit since the closed source libs are not distributed for 64bit. Nobody knows
why).

I'll attach override.c in a minute.

You can also use this to debug SLVoice in gdb, although due to slow start
up or otherwise, I've not managed to get this working directly: the first time
gdb starts I get an error "Failed to create texture". However, since that happens
even before the viewer connects to the daemon you can then simply exit
gdb and run the gdb command from the command line, it will pick up
where it left (get the connection and run).

In this case I use the following /usr/games/SLVoice script:

#! /bin/sh
gnome-terminal --geometry=165x24+0+0 -e "gdb -x /usr/games/gdbcmds /usr/games/SLVoice-bin"

where /usr/games/gdbcmds contains for example:

set args -p tcp -h -c -ll -1
set env LD_LIBRARY_PATH /usr/local/install/glibc-2.7/lib32:/usr/local/lib32:/lib32:/usr/lib32
set env LD_PRELOAD /usr/games/liboverride.so
b main
r

(where /usr/local/install/glibc-2.7/lib32 contains debugable libc in my case).

PS The attached override.c was written to work on my 64bit linux OS. It is not possible
to intercept 'open' (the sys call) because once inside libc it prefends one from hijjacking
functions by not using the PLT. Therefore one has to intercept the very first function
of libc that is called from outside libc, fopen64 in this case. I'm not 100% sure what
would be the correct function on a 32bit linux (though I wouldn't be surprised if it
is the same fopen64).


Carjay McGinnis added a comment - 16/Nov/08 08:51 AM
According to the Vivox mailing list a fixed version is available in the FirstLook Viewer for SLim.

It's of no use for the regular viewer though since the protocol obviously changed a bit so it won't work if you try to use it with a release viewer.


Rob Linden added a comment - 18/Nov/08 05:30 PM
We should probably consolidate the conversation and watchlists into VWR-4589, since as near as I can tell, this is a duplicate of that issue. I really like the workaround presented here better than the "custom kernel" solution, though.

Xaden Kjeller added a comment - 05/Feb/09 02:49 PM
This issue has finally been resolved in the latest release candidate for the 1.22 viewer (1.22.8.109366). Thanks to everyone who voted / commented / worked to get this issue fixed.