Why Niche SAP Integration Tools Still Drag Down Performance

In a previous post, we talked about why iPaaS, ETL, and middleware tools are challenged by the poor API experience of using SAP NetWeaver Gateway when it comes time to efficiently build integrations into SAP ERP from various front ends.

Beyond these generalists, there are some niche integration providers who attempt to overcome the complexity by pre-building integration libraries that are simpler to use because they target a single front end (like Salesforce) or category (like eCommerce). In particular, these vendors try to supply “mostly” pre-built toolsets that handle the specific functionality needed to connect SAP ERP to their front end of choice.

These tools have many of the same challenges and limitations as the integration platforms when trying to connect to SAP, but they are at least able to reduce the complexity for customers by doing much of the hard work of integration building. They can do this effectively because they only have to worry about specific pieces of data on the SAP side that map to whatever their front-end target needs to do its job.

These vendors universally use SAP’s JCo connector to call the same publicly available SAP ERP BAPIs and RFCs, which allows them to get up and running without needing to touch SAP. This approach is similar to the samples or templates provided by iPaaS vendors for SAP integration, which offer starting points to address certain use cases.

But this architecture also means that the ability to handle custom business logic or data must be built in their tool. As a result, implementation time must be spent trying to determine what that business logic is for any particular customer’s SAP instance and then reproducing it in their proprietary system. Just as important, this approach has negative performance implications — both per query as well as in increasing aggregate load on the SAP instance (impairing performance across all other SAP usage). Let’s dig in to understand why.

JCo? You must be joking

A core piece of most integrations to SAP ERP is the JCo connector. This is a piece of software (written in Java — hence the name, which is short for “Java Connector”) typically run on a standalone server as part of the integration architecture. So, what’s the big deal?

To start with, we now have to set up and maintain this additional server. Additionally, it is renowned for random, memory-related crashes (multiples times a month is not uncommon). This means this server needs to be continually monitored and reset whenever it goes down — which, according to Murphy’s Law, will always occur during your periods of highest usage and won’t be noticed for some time before being corrected.

Passing every call through this server adds additional latency to your integration processes. And because the JCo version is tightly tied to the version of NetWeaver in use, it needs to be updated whenever the ERP is updated (and vice versa), leading to extra maintenance, and, in some cases, an inability to keep up with the latest version of the connector due to an inability to upgrade the ERP version.

Take it outside

Beyond the JCo connector, there are significant performance implications when trying to do any sort of complex operation in SAP from the outside. A variety of issues come into play here, including:

• Not all RFCs are remotely accessible: This is especially true in industry add-ons, meaning you have to reproduce their logic on your own in the front end.

• The available inquiry (search) calls are extremely limited: For example, they don’t handle custom fields. They may also require data (like sales organization or plant) that isn’t technically needed to execute the search you want and that you don’t have easily available in the context where you are trying to do the search.

• Almost every operation requires multiple function calls: This often requires session management in the front end to ensure that the function calls take place in the same SAP session. And even then, it can be difficult to call BAPIs asynchronously while still ensuring proper execution order without handling it explicitly in the calling application

But perhaps most significantly, for security reasons, SAP provides no functionality available from outside the ERP to execute arbitrary SQL calls (for example joins between tables), and single-table reads have extreme limitations. In particular, these queries have poor error handling, the filters are typically limited to just 75 characters in length, and each row returned cannot contain more than 512 characters (across all fields). It’s important to understand the implications of this because it severely handicaps any attempt to make effective use of SAP ERP data when accessing it from the outside.

A very simple example of a very big problem

Suppose we want to find the first 10 orders last month that included part #1234. Let’s pretend for the sake of simplicity that there are two tables in SAP that contain this information: a table of orders that include the order number and date, and a table of line items where each row includes the order number and the part number for that line item, among other information.

If we’re doing it inside SAP, this is a trivial task — we can execute a database query that combines these two tables, filters by part number and orders from last month, sorts the results by date, and then finds the first 10 rows — all in one query. This query will be executed in the database and is extremely efficient because the database is designed to do exactly this sort of work.   

But outside of SAP, we can’t do this because of the inability to join multiple tables. So, we have to do the work in our application, which we might represent as follows:

Because the order table doesn’t know which orders have part #1234, we have to pull the order header information (order number and sale date) for every order last month — probably sorted by date to make our life easier later. Note that if we wanted to pull more fields than just the order number and date, we’d quickly run into the data limits on query results and would have to do multiple calls to get each row (piecing them together in our application).

Then, because the line-item table doesn’t know what date each line item was for, we have to pull the order number for every line-item detail that has part #1234. We then filter the list of orders in the first step to only include order numbers that also show up in this second step. We now have a list of orders from last month that include the correct part (and it’s sorted by date if we were smart above). So, now we just find the first 10 orders.

Remember:  This is simple! Imagine trying to do a search that needs data from 10 or more tables. This isn’t unusual — there are 20 to 30 thousand tables in a typical SAP ERP implementation, and something like a quote often involves combining 30 or more tables to get all the needed data.

Keep it simple

The same request from enosix consists of a single API call that does all its work in SAP, taking advantage of the power and speed of that database engine, and then returns simply the 10 rows of results. By contrast, to do the same call outside takes at least two (and in practice many more) API calls, each of which might return thousands or millions of rows.

Joining and filtering these rows in a front-end application will always be significantly slower than doing it in a database designed for this work. All this complexity dramatically increases the bandwidth needs and the load on the SAP server for every API call — which adds up quickly when you’re running a high-volume application.

As a result, many platforms find it’s often easier to just replicate all of this data into the front-end so the application can take advantage of the speed of its own database to produce the results. But then we’ve lost the power of real-time data!

Who wants a one-trick pony?

All of these issues are why you’ll find notes like this one from Jitterbit’s user documentation’s best practices on using their SAP connector:

“The integration is much more challenging (and becomes a project risk factor) if the integration is bidirectional. In other words, SAP Customers flow to the target endpoint, and then any new Customers and Partner Functions have to be created in SAP from that target endpoint. As SAP’s object creation data validation rules are stricter than other endpoints, this requires considerable custom work on both the endpoint and SAP itself to handle the Partner Functions.”

This isn’t Jitterbit-specific but rather a limitation of using the JCo connector in this way — which every other integration provider does.

And, of course, even setting aside the performance and reliability issues, most of these templates and tools only support a single front end. What happens when you want to add a second use case, such as expanding from CRM to eCommerce or field service? Or for that matter, what if you decide to use a different application vendor — for example, changing from Salesforce to a different CRM or adding a homegrown tool for some special need?

In any of these cases, you’re going to be left starting over – not just needing to build another integration, but needing to find a whole new platform to do it in.

Come to the jack of all trades (and master of all)

This is why, when we built the enosix framework, we created our own communication framework to avoid the JCo connector altogether, and we put all of the execution logic inside SAP. This way we can use the business logic that already lives there and the full power of the internal database and API layer to efficiently gather the needed data or execute the required transaction – while the front end is relegated to simply making a single, straightforward API call. 

If you need a flexible, high-volume integration platform for SAP ERP, enosix is really the only way to go. Schedule your demo today to see the difference for yourself.