Skip to content

Commit

Permalink
issue #13 - some fixes to get all examples working again
Browse files Browse the repository at this point in the history
  • Loading branch information
justb4 committed Sep 23, 2014
1 parent 5317e19 commit 6779b3d
Show file tree
Hide file tree
Showing 8 changed files with 141 additions and 333 deletions.
23 changes: 12 additions & 11 deletions docs/using.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,15 +59,16 @@ Configuring Components
----------------------
Most Stetl Components, i.e. inputs, filters, outputs, have properties that can be configured within their
respective [section] in the config file. But what are the possible properties, values and defaults?
This is documented within each Component class using the `@Config` decorator much similar to `@property`, only with
This is documented within each Component class using the ``@Config`` decorator much similar to the standard Python
``@property``, only with
some more intelligence for type conversions, defaults, required presence and documentation.
It is loosely based on https://wiki.python.org/moin/PythonDecoratorLibrary#Cached_Properties and Bruce Eckel's
http://www.artima.com/weblogs/viewpost.jsp?thread=240845 with a fix/hack for Sphinx documentation.

See for example the :class:`stetl.inputs.fileinput.FileInput` documentation.

For class authors: this information is added
via the Python Decorators much similar to `@property`. The :class:`stetl.component.Config`
via the Python Decorators much similar to ``@property``. The :class:`stetl.component.Config`
is used to define read-only properties for each Component instance. For example, ::

class FileInput(Input):
Expand Down Expand Up @@ -112,8 +113,8 @@ is used to define read-only properties for each Component instance. For example,
self.file_list = Util.make_file_list(self.file_path, None, self.filename_pattern, self.depth_search)

This defines two configurable properties for the class FileInput.
Each ``@Config`` has three parameters: ``p_type``, the Python type (``str``, ``list``, ``dict``, ``bool``,``int``),
`default` (default value if not present) and `required` (if property in mandatory or optional).
Each ``@Config`` has three parameters: ``p_type``, the Python type (``str``, ``list``, ``dict``, ``bool``, ``int``),
``default`` (default value if not present) and ``required`` (if property in mandatory or optional).

Within the config one can set specific
config values like, ::
Expand All @@ -122,23 +123,23 @@ config values like, ::
class = inputs.fileinput.XmlFileInput
file_path = input/cities.xml

This automagically assigns `file_path` to `self.file_path` without any custom code and assigns the
default value to `filename_pattern`. Automatic checks are performed: if `file_path` (``required=True``) is present, if its type is string.
In some cases type conversions may be applied e.g. when type is `dict` or `list`. It is guarded that the value is not
overwritten and the docstrings will appear in the auto-generated documentation each property with a ``CONFIG`` tag.
This automagically assigns ``file_path`` to ``self.file_path`` without any custom code and assigns the
default value to ``filename_pattern``. Automatic checks are performed: if ``file_path`` (``required=True``) is present, if its type is string.
In some cases type conversions may be applied e.g. when type is ``dict`` or ``list``. It is guarded that the value is not
overwritten and the docstrings will appear in the auto-generated documentation, each entry prepended with a ``CONFIG`` tag.

Running Stetl
-------------

The above ETL spec can be found in the file `etl.cfg`. Now Stetl can be run, simply by typing ::
The above ETL spec can be found in the file ``etl.cfg``. Now Stetl can be run, simply by typing ::

stetl -c etl.cfg

Stetl will parse `etl.cfg`, create all Components by their class name and link them in a Chain and execute
Stetl will parse ``etl.cfg``, create all Components by their class name and link them in a Chain and execute
that Chain. Of course this example is very trivial, as we could just call XSLT without Stetl. But it becomes interesting
with more complex transformations.

Suppose we want to convert the resulting GML to an ESRI Shapefile. As we cannot use GDAL `ogr2ogr` on the input
Suppose we want to convert the resulting GML to an `ESRI Shapefile`. As we cannot use GDAL ``ogr2ogr`` on the input
file, we need to combine XSLT and `ogr2ogr`. See example
`3_shape <https://github.com/justb4/stetl/tree/master/examples/basics/3_shape>`_. Now we replace the output
by using `outputs.ogroutput.Ogr2OgrOutput`, which can execute any `ogr2ogr` command, converting
Expand Down
10 changes: 5 additions & 5 deletions examples/bgt/etl-bgt-postgis-anonymous.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@
#input_file="input/20130814_GML_crotec_input.gml"
input_file="input/Maastricht_GML_20130924.gml"

# Databasse connection
# Database connection
host="localhost"
port="5432"
user="myusername"
database="mydatabasename"
password="mypassword"
user="postgres"
database="bgt"
password="postgres"
schema="public"

stetl -c etl-bgt-postgis.cfg -a "gml_input=$input_file host=$host port=$port user=$user database=$database password=$password schema=$schema max_in_memory_features=100000"
stetl -c etl-bgt-postgis.cfg -a "gml_input=$input_file host=$host port=$port user=$user database=$database password=$password schema=$schema max_in_memory_features=10000"

2 changes: 1 addition & 1 deletion examples/bgt/etl-bgt-shp.sh
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ input_file="input/Maastricht_GML_20130924.gml"
output_file="output/bgt-gml-test.shp"
# output_file="/Users/just/project/stetl/contrib/duiv/data/20130814_GML_crotec_output.gml"

stetl -c etl-bgt-shp.cfg -a "gml_input=$input_file shp_output=$output_file max_in_memory_features=100000"
stetl -c etl-bgt-shp.cfg -a "gml_input=$input_file shp_output=$output_file max_in_memory_features=10000"

4 changes: 4 additions & 0 deletions examples/inspire/ad-bag-nl/readme.txt
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@ at the INSPIRE-FOSS project:

https://code.google.com/p/inspire-foss/source/browse/#svn%2Ftrunk%2Fetl%2FNL.Kadaster%2FAddresses

NEW (23.9.2014)
See also the examples/basics/10_jinja2_templating for a more compact INSPIRE example.


5 changes: 4 additions & 1 deletion examples/ordnancesurvey/etl.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,7 @@ PYTHONPATH=.:$PYTHONPATH
GML_FILES=input
# GML_FILES=/Users/just/project/stetl/contrib/astun/osdata/58116-SX9192-2c1.gml

stetl -c stetl.cfg -a "database=ordsurvey host=localhost port=5432 user=postgres password=postgres schema=osmm temp_dir=temp max_features=5000 gml_files=$GML_FILES"
stetl=stetl
# stetl=../../stetl/main.py

$stetl -c stetl.cfg -a "database=ordsurvey host=localhost port=5432 user=postgres password=postgres schema=osmm temp_dir=temp max_features=5000 gml_files=$GML_FILES"
Loading

0 comments on commit 6779b3d

Please sign in to comment.