Wikix: Difference between revisions

(21 intermediate revisions by 13 users not shown)

Line 1:

'''Wikix''' is a 'C' based program written by Jeffrey Vernon Merkey that will read any XML dump provided by the foundation, extract all image names from the XML dump which it may reference, then generate a series of BASH or Bourne Unix style scripts which can be invoked to download all images from ~~Wikipedia~~ Commons and Wikipedia.

'''Wikix''' is a 'C' based program written by Jeffrey Vernon Merkey that will read any XML dump provided by the foundation, extract all image names from the XML dump which it may reference, then generate a series of BASH or Bourne Unix style scripts which can be invoked to download all images from ''Wikimedia Commons'' and ''Wikipedia''.

The program relies on [[:en:cURL|cURL]], an automated web spider, to download referenced images. The program will also convert text based utf8 characters into actual utf8 strings for those dumps which may contain improperly formatted names for specific images. The program can be configured to generate 16 parallel scripts which will download all images from Wikipedia. The program includes Jeff Bezanson's utf8 libraries.

~~The program relies on Curl~~, ~~an automated web spyder~~, ~~to download referenced images. The program will also convert text based utf8 characters into actual utf8 strings for those dumps which may contain improperly formatted names for specific~~ images~~. The program~~ can be ~~configured to generate 16 parallel scripts which will download all images from Wikipedia~~. ~~The program includes Jeff Bezanson's utf8 libraries.~~

As of March 24, 2008, using a cable modem, the entire set of Wikipedia images can be downloaded in about 96 hours using this program (420 GB as of 3/24/08).

On ~~a cable modem~~, the ~~entire set~~ of ~~Wikipedia images can be downloaded~~ in ~~about 36 hours using~~ this ~~program~~.

== Compiling and Installation ==

First download the source code from the [[Wikix#Source_Code|Source Code]] section.

=== On Ubuntu ===

#Extract the contents of ''wikix.tar.gz''. Suppose the source code is extracted into ''/home/you/wikix''.

#Start your terminal program. e.g. ''Konsole'' (in KDE)

#You need to install some packages before you compile Wikix. Type in your terminal:-<br><pre>sudo aptitude install libssl-dev build-essential curl</pre>

#Now goto the directory that contains the extracted source code, e.g. ''/home/you/wikix'', by typing<br><pre>cd /home/you/wikix</pre>

#Now type in your terminal.<br><pre>make</pre>Now if the compilation and linking of Wikix completes without errors then you will have a brand new executable - ''wikix'', in your ''/home/you/wikix'' (in this example) directory.

In case of any problems, please report it in the [[Talk:Wikix|discussion]] page.

== Options ==

<pre>

# ./wikix -h

USAGE: wikix -htrciop < file.xml [ > script.out ]

-h this help screen

-t use xml dump to strip from tree

-r wikipedia path

-c commons path

-i image path

-o output path

-p parallel (16 process) mode

</pre>

== Example ==

The program would typically be invoked in a directory that you wish to use to host the images. Wikix will construct a MediaWiki style directory structure which can be quickly imported into a MediaWiki Wikipedia installation:

The program would typically be invoked in a directory that you wish to use to host the images. Wikix will construct a MediaWiki style directory structure which can be quickly imported into a MediaWiki Wikipedia installation (e.g., via <tt>php rebuildImages.php --missing</tt>):

<pre>

wikix -p < name_of_xml_file.xml &

wikix -p < name_of_xml_file.xml > script.out &

</pre>

The -p option tells wikix to create parallel scripts. If you omit the -p option, it will create one very large file. By default, the program is ~~setup~~ to mirror the English Wikipedia. You can override the default settings by substituting path information for commons and the target Wikipedia site through the command line options.

The -p option tells wikix to create parallel scripts. If you omit the -p option, it will create one very large file. By default, the program is set up to mirror the English Wikipedia. You can override the default settings by substituting path information for commons and the target Wikipedia site through the command line options.

The program will create a series of scripts as:

Line 38:

Line 63:

To start the download, simply type

<pre>

$./image_sh

</pre>

In case you get the following error.

<pre>

./image_sh: <line no.>: Syntax error: Bad fd number

</pre>

Then please open the <tt>image_sh</tt> all the <tt>imagexx</tt> files and change the topmost line from

<pre>

#!/bin/sh

</pre>

to

<pre>

#!/bin/bash

</pre>

then re-type

<pre>

Line 45:

Line 94:

== Source Code ==

The full source code with build scripts can be download from this ftp server:

*ftp://www.wolfmountaingroup.org/pub/wikix/wikix.tar.gz

=== Source Code Files ===

'''Makefile'''

@@ Line 1: / Line 1: @@
-'''Wikix''' is a 'C' based program written by Jeffrey Vernon Merkey that will read any XML dump provided by the foundation, extract all image names from the XML dump which it may reference, then generate a series of BASH or Bourne Unix style scripts which can be invoked to download all images from Wikipedia Commons and Wikipedia.
+'''Wikix''' is a 'C' based program written by Jeffrey Vernon Merkey that will read any XML dump provided by the foundation, extract all image names from the XML dump which it may reference, then generate a series of BASH or Bourne Unix style scripts which can be invoked to download all images from ''Wikimedia Commons'' and ''Wikipedia''.
+{{?}}
+The program relies on [[:en:cURL|cURL]], an automated web spider, to download referenced images.  The program will also convert text based utf8 characters into actual utf8 strings for those dumps which may contain improperly formatted names for specific images.  The program can be configured to generate 16 parallel scripts which will download all images from Wikipedia.  The program includes Jeff Bezanson's utf8 libraries.
-The program relies on Curl, an automated web spyder, to download referenced images.  The program will also convert text based utf8 characters into actual utf8 strings for those dumps which may contain improperly formatted names for specific images.  The program can be configured to generate 16 parallel scripts which will download all images from Wikipedia.  The program includes Jeff Bezanson's utf8 libraries.
+As of March 24, 2008, using a cable modem, the entire set of Wikipedia images can be downloaded in about 96 hours using this program (420 GB as of 3/24/08).
-On a cable modem, the entire set of Wikipedia images can be downloaded in about 36 hours using this program.
+== Compiling and Installation ==
+First download the source code from the [[Wikix#Source_Code|Source Code]] section.
+=== On Ubuntu ===
+#Extract the contents of ''wikix.tar.gz''. Suppose the source code is extracted into ''/home/you/wikix''.
+#Start your terminal program. e.g. ''Konsole'' (in KDE)
+#You need to install some packages before you compile Wikix. Type in your terminal:-<br><pre>sudo aptitude install libssl-dev build-essential curl</pre>
+#Now goto the directory that contains the extracted source code, e.g. ''/home/you/wikix'', by typing<br><pre>cd /home/you/wikix</pre>
+#Now type in your terminal.<br><pre>make</pre>Now if the compilation and linking of Wikix completes without errors then you will have a brand new executable - ''wikix'', in your ''/home/you/wikix'' (in this example) directory.
+In case of any problems, please report it in the [[Talk:Wikix|discussion]] page.
+== Options ==
+<pre>
+# ./wikix -h
+USAGE:  wikix -htrciop < file.xml [ > script.out ]
+              -h   this help screen
+              -t   use xml dump to strip from tree
+              -r   wikipedia path
+              -c   commons path
+              -i   image path
+              -o   output path
+              -p   parallel (16 process) mode
+</pre>
 == Example ==
-The program would typically be invoked in a directory that you wish to use to host the images.  Wikix will construct a MediaWiki style directory structure which can be quickly imported into a MediaWiki Wikipedia installation:
+The program would typically be invoked in a directory that you wish to use to host the images.  Wikix will construct a MediaWiki style directory structure which can be quickly imported into a MediaWiki Wikipedia installation (e.g., via <tt>php rebuildImages.php --missing</tt>):
 <pre>
-wikix -p < name_of_xml_file.xml &
+wikix -p < name_of_xml_file.xml > script.out &
 </pre>
-The -p option tells wikix to create parallel scripts.  If you omit the -p option, it will create one very large file.  By default, the program is setup to mirror the English Wikipedia.  You can override the default settings by substituting path information for commons and the target Wikipedia site through the command line options.
+The -p option tells wikix to create parallel scripts.  If you omit the -p option, it will create one very large file.  By default, the program is set up to mirror the English Wikipedia.  You can override the default settings by substituting path information for commons and the target Wikipedia site through the command line options.
 The program will create a series of scripts as:
@@ Line 38: / Line 63: @@
 To start the download, simply type
+<pre>
+$./image_sh
+</pre>
+In case you get the following error.
+<pre>
+./image_sh: <line no.>: Syntax error: Bad fd number
+</pre>
+Then please open the <tt>image_sh</tt> all the <tt>imagexx</tt> files and change the topmost line from
+<pre>
+#!/bin/sh
+</pre>
+to
+<pre>
+#!/bin/bash
+</pre>
+then re-type
 <pre>
@@ Line 45: / Line 94: @@
 == Source Code ==
+The full source code with build scripts can be download from this ftp server:
+*ftp://www.wolfmountaingroup.org/pub/wikix/wikix.tar.gz
+=== Source Code Files ===
 '''Makefile'''

Wikix: Difference between revisions

wikademia>Wikademia