Speech:Spring 2014 Justin Alix Log


 * Home
 * Semesters
 * Spring 2014
 * Proposal
 * Report
 * Information - General Project Information
 * Experiments - List of speech experiments

Week Ending February 4th, 2014
1/31 - Review logs from previous Tools groups to determine how the tools have progressed over previous semesters. Review other group logs from previous semesters to gain a better understanding of where the overall project is currently at.
 * Task:

2/2 - Log into Caesar, change my password, and look at the servers file systems

2/3 - Log into our server and browse the file system, continue to read previous logs, research tools.

2/5 - Finish and upload Proposal 1/31 - I'd say I have a good understanding of where the project is currently, as well as the tools involved in doing so. I've read up quite a bit on speech recognition in general and it is very interesting.
 * Results:

2/2 - After logging into the system It was cake to change my password (as it should be). Browsing the file system provided an understanding of the structure, as well as where several components reside (such as experiments and programs)

2/3 - Same as above, and I did some heavy research on the updated version of the tools used in the system (Sphinx, Dictionary, etc). To my surprise a few are not supported anymore, and many of the tools are quite outdated, but they cannot be updated due to dependencies and the use of openSUSE 11.3.

2/5- The proposal has been updated.

Gain an accurate understanding of the project as a whole, the current system setup and structure, and the status of the several tools in use. That the system cannot be updated without a huge overhaul of almost all components. The cost/benefits will have to be considered, especially since (as mentioned in class) the end of the research work is nearing.
 * Plan:
 * Concerns:

Week Ending February 11, 2014
2/6 - Attempt to install openSUSE 11.3 on my pc. This will reduce the need to SSH into our server and allows me to play around a bit more than I would be comfortable with on Trib. I want to try to get a replication of the Speech Recog. software going, so I'm going to install all the same tools and see if I can pull some of the data we use. This will also allow me to mess with upgrading the tool components freely and more comfortably.
 * Task:

2/11 - openSUSE is installed and running, testing of the upgraded tools has begun (specifically Sphinx 4.0).

2/6 - The VM is up and running, although the tools have not been installed yet. The ISO of 11.3 was a little hard to find, but I have a copy on my local HDD.
 * Results:

2/11 - Sphinx requires JDK6, among other software. The install of the latest JDK for openSUSE was completed, and is rather easy. I am currently running into problems installing ANT, a requirement for S4. UPDATE: ANT 1.7 (or less) is not available, and 1.8-1.9 require openSUSE 12.2-13.1 for their respective versions (http://software.opensuse.org/package/ant-apache-oro), so a discussion with the systems group is in order. As of now it looks like we may not be able to upgrade, or it will not be worth it with S4 only providing a .221 decrease in error rate compared to S3.

2/11 - Java Service Wrapper requires quite a bit more time to understand, so I am moving on to check SCLite integration. UPDATE: SCLITE v 2.9 has been installed on my VM. It should be able tow work on Caesar without error.

2/16 - Wrote in the group log and the proposal log.
 * Plan:

All of the tools are outdated and require a pretty large overhaul of the entire system. Since the semester is already up and running and tests are being run I think it may not be worth it at this point. We will have to talk to the systems group and see what we can accomplish, and in what amount of time, if we are to upgrade everything.
 * Concerns:

Week Ending February 18, 2014
2/13 - Upon Prof. Jonas's recommendation I will set up a VM with Fedora to test our tools on. Rome, the Dell tower n the server room, is being used by the System team to see how Fedora would handle trains, and how they would compare to those run on openSUSE. By creating a VM I can evaluate the tools on Fedora 19 and how a possible system wide upgrade would work (switching from openSUSE to Fedora would allow for a much more recent and supported OS).
 * Task:

2/13 - Updated the proposal.


 * Results:


 * Plan:


 * Concerns:

Week Ending February 25, 2014
2/20 - Read other peoples logs today, to find out where they are with their work. I am reading the JSW documentation as well and seeing if I can wrap my head around it.
 * Task:

2/21 - Attempt to get the tools running on a Fedora machine. I have been working on this for 4~ hours. I'll post results tomorrow in results. Harry Dodson's logs lead to a good tutorial on installing the system on a machine.

2/22 - Today a few hours were spent installing tools on my openSUSE VM, trying to get Sphinx3.0.8 to work. Several .h files were missing from the downloaded tar.gz. Some initial work with JSW was performed as well. Bego's logs from last semester have some good info on Sphinx 4.0, and I am quickly coming to the same conclusion - that it isn't worth it. More detailed results for today's work, as well as work on the 21st, will be posted tomorrow under results (busy weekend!).

2/22 - The installation of the tools on Fedora 19 is complete. It was not all smooth sailing, however. I tried to install Sphinx 3.7, and the system threw an error. After some research it was because SphinxBase was not installed (just like the name implies, SphinxBase installs the base requirements for much of the Sphinx software). Trying to install SphinxBase, I ran into several errors about missing software, so I installed them with the commands below
 * Results:

Using SU in terminal: yum install autoconf yum install libtool yum install automake

./autogen.sh would still not run, it then threw more errors about missing software, remedied by:

yum install bison yum install swig

autogen.sh and ./conf ig then worked. However, make would, not, I received an error for not having Python (which was only needed for the tutorial I was following. The scripts we use in Capstone are inPerl).

The error returned was: Python.h: No such file or directory

The solution was: yum install python-devel

I double checked the Python installation:

[root@localhost sphinxbase]# python --version Python 2.7.5

Now the MAKE command worked. Before installing, however, I wanted to check the installation using the make check: command. This was the result

- Testsuite summary for sphinxbase 0.8 - - See test/regression/test-suite.log - make[4]: *** [test-suite.log] Error 1 make[4]: Leaving directory `/usr/src/sphinxproject/sphinxbase/test/regression' make[3]: *** [check-TESTS] Error 2 make[3]: Leaving directory `/usr/src/sphinxproject/sphinxbase/test/regression' make[2]: *** [check-am] Error 2 make[2]: Leaving directory `/usr/src/sphinxproject/sphinxbase/test/regression' make[1]: *** [check-recursive] Error 1 make[1]: Leaving directory `/usr/src/sphinxproject/sphinxbase/test' make: *** [check-recursive] Error 1 -    sphinxbase 0.8: test/regression/test-suite.log - .. contents:: :depth: 2
 * 1) TOTAL: 10
 * 2) PASS:  5
 * 3) SKIP:  0
 * 4) XFAIL: 0
 * 5) FAIL:  5
 * 6) XPASS: 0
 * 7) ERROR: 0
 * 1) TOTAL: 10
 * 2) PASS:  5
 * 3) SKIP:  0
 * 4) XFAIL: 0
 * 5) FAIL:  5
 * 6) XPASS: 0
 * 7) ERROR: 0

FAIL: test-sphinx_fe-ctl.sh - WAVE2FEAT CTL/WAV/SPH TEST WAVE2FEAT test FAILED Sub-tests failed: WAVE2FEAT test FAIL: test-sphinx_fe-dither-seed.sh - WAVE2FEAT-DITHER-SEED TEST WAVE2FEAT-DITHER-SEED test FAILED Sub-tests failed: WAVE2FEAT-DITHER-SEED test FAIL: test-sphinx_fe-logspec.sh - WAVE2FEAT-LOGSPEC TEST WAVE2FEAT-LOGSPEC test FAILED Sub-tests failed: WAVE2FEAT-LOGSPEC test FAIL: test-sphinx_fe.sh - WAVE2FEAT TEST WAVE2FEAT test FAILED Sub-tests failed: WAVE2FEAT test FAIL: test-sphinx_fe-smoothspec.sh - WAVE2FEAT-SMOOTHSPEC TEST WAVE2FEAT-SMOOTHSPEC test FAILED Sub-tests failed: WAVE2FEAT-SMOOTHSPEC test -   sphinxbase 0.8: test/regression/test-suite.log - .. contents:: :depth: 2 FAIL: test-sphinx_fe-ctl.sh - WAVE2FEAT CTL/WAV/SPH TEST WAVE2FEAT test FAILED Sub-tests failed: WAVE2FEAT test FAIL: test-sphinx_fe-dither-seed.sh - WAVE2FEAT-DITHER-SEED TEST WAVE2FEAT-DITHER-SEED test FAILED Sub-tests failed: WAVE2FEAT-DITHER-SEED test FAIL: test-sphinx_fe-logspec.sh - WAVE2FEAT-LOGSPEC TEST WAVE2FEAT-LOGSPEC test FAILED Sub-tests failed: WAVE2FEAT-LOGSPEC test FAIL: test-sphinx_fe.sh - WAVE2FEAT TEST WAVE2FEAT test FAILED Sub-tests failed: WAVE2FEAT test FAIL: test-sphinx_fe-smoothspec.sh - WAVE2FEAT-SMOOTHSPEC TEST WAVE2FEAT-SMOOTHSPEC test FAILED Sub-tests failed: WAVE2FEAT-SMOOTHSPEC test
 * 1) TOTAL: 10
 * 2) PASS:  5
 * 3) SKIP:  0
 * 4) XFAIL: 0
 * 5) FAIL:  5
 * 6) XPASS: 0
 * 7) ERROR: 0

As you can see, 5 of the 10 make checks failed. The program was not compiling correctly, and I found this rather annoying. I decided to move on to the install of Sphinx 3.0.7 and see how it went. The commands as follows as SU:

./autogen.sh (performed twice) ./configure

An error was returned upon running ./configure:

../../../include/s3types.h:84:21: fatal error: libutil.h: No such file or directory

I moved back to Sphinxbase and decided to give it another go, this time with a clean install. I received the same errors. I found the reason was because I was not set as the owner of the file (apparently I wasn’t performing it as SU, but I was doing it under my normal name). So I set myself as the owner, and figured I would set ‘Justin’ as the owner of the whole /usr folder.

I tried again with a clean install, and this time did not do make clean, and it worked.

With SphinxBase installed, I moves back to Sphinx3.0.7 ./autogen.sj ./configure make fatal error: libutil.h: No such file or directory
 * 1) include 

The same error. I decided to turn to other peoples logs. Harry Dodson from last semester had great links to setting up all the software. Unfortunately, setting ‘Justin’ as the owner was a BIG MISTAKE. That removed the /usr ownership by root, which apparently terminates the root password as well. So I basically blew up my virtual machine (I could no longer use root).

After re-installing Fedora 19 I tried a fresh install of SphinxBase and Sphinx3. Now, of course, they work. There are a few tools left to install, then I can try to run a train on a Fedora machine and see how it goes.

2/25 -

With Sphinxbase and Sphinx 3 installed on the Fedora VM, I wanted to complete the full software installation. After booting it up it was back to terminal su cp /sphinxtrain make All went well. On to the toolkit, which was downloaded from sourceforge and extracted. ./configure make make install Everything installed well. Now that the software is all set up, all thats left to do is see how it handles a train and experiment. I've decided to hold off on these, though, as I want to work on getting everything running on the openSUSE VM. So off I went to my openSUSE VM, and into the terminal cd /sphinx 3.0.8 ./autogen .sh An error was recieved libutil.h: No such file or directory So I decided to try a fresh install. cd /sphinx3 ./autogen ./configure Returned an error: prim_type.h: No such file or directory So I went off to install Sphinxbase, taking care of the dependancy. cd .. cd /sphinxbase ./autogen.sh ./configure make make install Everything installed fine! So next I installed SphinxTrain following the same procedure cd /sphinxtrain ./autogen.sh ./configure make make install Everyhting installed fine, so i went back to Sphinx. I ran into the same error, libutil.h was not found. After copying the file from an online resource, along with about 10 other missing files, I consulted the internet, and it turns out Sphinx 3.0.8 is not compatible with SphinxBase0.8

As a result I uninstalled Sphinxbase0.8 and installed sphinxbase 0.7. I had to move the SphinxBase files to the Sphinx3 directory, and ran ./configure with a parameter telling the system where the files were. It configured fine, but upon the make command, new errors occured. This was after about 3hrs of work, so I am now done with it. At least everything worked on Fedora. I'll continue on Sphinx4 and the JSW for now


 * Plan:


 * Concerns:

Week Ending March 4, 2014
2/28 - Read classmates logs and see what they are up to. 3/4 and 3/5 - continue with the JSW and get it working
 * Task:
 * Results:

3/4 -

Today’s focus was to make headway with the Java Service Wrapper. The latest version can be downloaded from: http://wrapper.tanukisoftware.com/doc/english/download.jsp The community edition is the free version, with no explicit license agreement or time limit (the normal versions are good for a one month trial). First, I downloaded and unpacked Sphinx4.1.0.6 betainto the usr/bin directory, and renamed it Sphinx4. Sphinx4 can be found at: http://sourceforge.net/projects/cmusphinx/files/sphinx4/1.0%20beta6/ After downloading and unpacking the JSW, I tried to run the testWrapper, which can be found on their website. http://wrapper.tanukisoftware.com/doc/english/example.html Unfortunately running the command ./testwrapper start

Did not start the service. Apparently there was a problem with initializing Java. An hour of research produced no apparent fix.

I followed the tutorial on the website, substituting the Boss app they use with Sphinx 4 http://wrapper.tanukisoftware.com/doc/english/integrate-simple-nix.html

First, please copy the following files into the JBoss bin directory: {WRAPPER_HOME}/bin/wrapper {WRAPPER_HOME}/src/bin/sh.script.in Rename the script file to reflect the name of the application. {v}/bin/Sphinx4 Now open the script into an editor. We need to set the long and short names to reflect that the script is being used to launch JBoss. You will see two variables immediately after the header of the script.APP_NAME and APP_LONG_NAME. Suggested values for these variables are shown below. APP_NAME="jboss" APP_LONG_NAME="JBoss Application Server"

Copy the native library into the JBoss bin/native/lib directory in case you plan to run 32-bit version of JBoss and the Wrapper or bin/native/lib64 directory for 64-bit: {WRAPPER_HOME}/lib/libwrapper.so

Finally you also need to copy the wrapper.jar file to JBoss's lib directory: {WRAPPER_HOME}/lib/wrapper.jar

The default configuration file wrapper.conf will place a wrapper.log file in a logs directory under the application's home directory. JBoss does not have such a directory by default, so we will need to create one. You should now have the following directory: {Sphinx4_HOME}/logs

I ran into a barrier, however, discerning the actual startup procedure for Sphinx4, which is the next step. I have decided to stop here, after a few hours of work.


 * Plan:


 * Concerns:

Week Ending March 18, 2014
3/15 - Checked other peoples logs to see what they have been up to.
 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending March 25, 2014
3/23 - Read through the new running a train script. 3/24 - Picked up the work with the Java Service Wrapper - I'll post results in the results section. 3/25 - Interestingly, here is a new sphinx version, 4.5 http://sourceforge.net/projects/cmusphinx/files/sphinx4/5%20prealpha/
 * Task:

3/23 - The main problem I was having earlier was this part of the documentation:
 * Results:

Locate the Application's Java Command Line

Before the Wrapper can be configured to launch an Application, you will need to know the full Java command which is normally used. Most applications make use of a script to build up the actual command line. These scripts tend to get quite unwieldy but in fact, the featured ability to avoid having to work with them is one of the benefits of working with the Wrapper. JBoss is launched by default using a script called run.sh. It is launched by first changing the current directory to the bin directory and then run from there. If you open run.sh into an editor, you will notice the following lines towards the end of the file: # Execute the JVM in the foreground eval \"$JAVA\" $JAVA_OPTS \ -Djava.endorsed.dirs=\"$JBOSS_ENDORSED_DIRS\" \ -classpath \"$JBOSS_CLASSPATH\" \ org.jboss.Main "$@" JBOSS_STATUS=$? The majority of the script has the task of collecting system specific information and storing that information into environment variables. The lines above then expands all of the collected information into the final Java command which launches the application. From looking at the source of the script, we hope you appreciate the complexity and the desire to have to avoid completely writing such scripts yourself. In order to configure the Wrapper, all that is really needed is the final expanded command line. Rather than reading through the entire script and attempting to understand it, we will use a simple trick to display the final command line in the console. Edit the script run.sh by changing it as follows: # Execute the JVM in the foreground echo eval \"$JAVA\" $JAVA_OPTS \ -Djava.endorsed.dirs=\"$JBOSS_ENDORSED_DIRS\" \ -classpath \"$JBOSS_CLASSPATH\" \ org.jboss.Main "$@" If you now rerun the script, you will see something like the following in the console (Your output will all be on one line): eval "/usr/lib/jvm/java-6-openjdk/bin/java" -server -Xms128m -Xmx512m -XX:MaxPermSize=256m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true -Dprogram.name=run.sh -Djava.library.path=/usr/lib/jboss-6.0.0.Final/bin/native/lib -Djava.endorsed.dirs="/usr/lib/jboss-6.0.0.Final/lib/endorsed" -classpath "/usr/lib/jboss-6.0.0.Final/bin/run.jar:/usr/lib/jvm/java-6-openjdk/lib/tools.jar" org.jboss.Main

As I understand, Sphinx4 does not have a straight Java command to run, it must be built first:

Building Sphinx-4

Since the sphinx4-{version}-bin.zip distribution does not contain the source code, you must download the sphinx4-{version}-src.zip, or retrieved the code from SourceForge using svn, in order to be able to build from the sources. The software required for building Sphinx-4 are listed in the Required Software section.

Setup JSAPI 1.0

Before you build Sphinx-4, it is important to setup your environment to support the Java Speech API (JSAPI), because a number of tests and demos rely on having JSAPI installed.

Run ant

To build Sphinx-4, at the command prompt change to the directory where you installed Sphinx-4 (usually, a simple “cd sphinx4” will do). Set required environment variables. JAVA_HOME to the location of JDK, ANT_HOME to the location of ant and and PATH to include both bin subfolder of JDK and bin subfolder of ant variables. For example:

export JAVA_HOME=/usr/local/jdk1.6.0_14 export ANT_HOME=/usr/local/apache-ant-1.8.0 export PATH=/usr/local/jdk1.6.0_10/bin:/usr/local/apache-ant-1.8.0/bin:$PATH Then type the following:

ant This executes the Apache Ant command to build the Sphinx-4 classes under the bld directory, the jar files under the lib directory, and the demo jar files under the bin directory.

To delete all the output from the build to give you a fresh start:

ant clean Create Javadocs

The javadocs have already been built if you downloaded the sphinx4-{version}-bin.zip. In order to build the javadocs yourself, you must download the sphinx4- {version}-src.zip distribution instead. To build the javadocs, go to the top level directory (“sphinx4-{version}”), and type:

ant javadoc This will build javadocs from public classes, displaying only the public methods and fields. In general, this is all the information you will need. If you need more details, such as private or protected classes, you can generate the corresponding javadoc by doing, for example:

ant -Daccess=private javadoc

This will pose a problem when performing this on the actual Caesar machine, as these dependencies (and and java docs) are the reason I am looking into a JW at all. As a result this can probably all be done on a different machine, then packaged to be put on the server.

Ant and Javadocs have been downloaded.

3/25 - As expected there is virtually no documentation on Sphinx 4-5


 * Plan:


 * Concerns:

Week Ending April 1, 2014
3/29 - Continued working with Sphinx4.
 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 8, 2014

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 15, 2014

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 22, 2014

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending April 29, 2014

 * Task:


 * Results:


 * Plan:


 * Concerns:

Week Ending May 6, 2014

 * Task:


 * Results:


 * Plan:


 * Concerns: