[Stackoverflow] [Progress OpenEdge ABL] Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK

[Justin Cohen]

Does anyone have experience developing SQL singer-taps using the Meltano SDK?

Problem​

My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap.

Context​

• I am developing in an Ubuntu environment running on a windows machine using WSL2.
• I have successfully established connection to the source using the custom sqlalchemy dialect I found for Progress Openedge (view here) on the ubuntu machine with a url created using this code:

pyodbc_connstr = urllib.parse.quote_plus(r"""
    DRIVER={DataDirect 7.1 OpenEdge Wire Protocol};
    HOST=xxxx;
    PORT=xxxx;
    DB=xxxxx;
    UID=sysprogress;
    PWD=;
    DEFAULTSCHEMA=PUB;
""")                                         

sa_str = "progress+pyodbc:///?odbc_connect={}".format(pyodbc_connstr)

• I am referring to this meltano article as guidance for setting up the project.
• I've set up the project with the following responses to the cookie cutter prompts:
  • Stream Type = 3 - SQL
  • Auth Type = 6 - Custom or N/A
• I've installed the progress_sa SQLAlchemy dialect into the project from the local directory on the Ubuntu machine using poetry add <local path to package>
• I've created a .env file for testing the tap.
• All developer TODO areas of the SDK are complete.

Where I'm stuck​

When running poetry run tap-<my tap> --config= ENV I get the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 493, in invoke
    tap = cls(
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 625, in __init__
    super().__init__(*args, **kwargs)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 102, in __init__
    self.setup_mapper()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 172, in setup_mapper
    self.mapper.register_raw_streams_from_catalog(self.catalog)
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 165, in catalog
    self._catalog = self.input_catalog or self._singer_catalog
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 318, in _singer_catalog
    for stream in self.streams.values()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 127, in streams
    for stream in self.load_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 351, in load_streams
    for stream in self.discover_streams():
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 655, in discover_streams
    for catalog_entry in self.catalog_dict["streams"]:
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/tap_base.py", line 643, in catalog_dict
    result["streams"].extend(connector.discover_catalog_entries())
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 497, in discover_catalog_entries
    engine = self._engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 305, in _engine
    self._cached_engine = self.create_engine()
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/singer_sdk/connectors/sql.py", line 322, in create_engine
    return sqlalchemy.create_engine(
  File "<string>", line 2, in create_engine
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
  File "/home/jcohen/tap-dev/tap-progressopenedge/.venv/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 680, in create_engine
    raise TypeError(
TypeError: Invalid argument(s) 'json_serializer','json_deserializer' sent to create_engine(), using configuration Progress_pyodbc/QueuePool/Engine.  Please check that the keyword arguments are appropriate for this combination of components.

I admittedly am still a developing python programmer, and so I am still learning how to effectively troubleshoot.

It is hard to tell if the issue is caused by something that I've done in setting up the tap or not.

Any help or advice would be greatly appreciated.

Continue reading...