[Sergio828]

I know very little about coding but I'm finding this thread fascinating to follow. Based on my experience with tuners, I've found very little appetite to fix lingering issues or provide much, if any, support for this platform. I'm running an emissions on tune currently, and while it is OK, there are some areas for improvement.

Thanks for detailing your adventure, will be watching!

 

[NavyCuda]

Last few days have been hectic. Today I managed to put in a couple hours since the last time I posted.

Mapping the winols to ecmParameter first has proven to be hugely beneficial. In two hours I'm close to halfway ready to start testing my converter. Being able to see how the data is supposed to flow really makes it easier to write the methods. I'm rather disappointed in myself I didn't do it sooner.

I'm hoping to get a few hours in tomorrow, ideally get to the point of testing the map converter and if successful I can go back to applying a complete configuration against the calibration.

I'm sure I've covered it before but a quick recap:

A Calibration is the file that contains the data that will physically be installed to the engine computer. A Configuration is a file that contains the details about each of the parameters stored in the table of contents. The table of contents is an area of the calibration that lists all of the engine parameters.

The engine parameters are stored in the ecm as a binary, the calibration stores the binary as hexadecimal, the configuration provides the name, description and correct conversions for the datatype that parameter represents.

 

[NavyCuda]

I didn't need my sanity, right?! he. hehehe.

So, I wrote a whole bunch of code and to my surprise it only had a couple minor bugs but otherwise worked as intended. The problems associated with mismatched calibrations and maps are starting to show their ugly head again. So far I have been able to extract over half of the x-axis itns from the y axis and verify them with the table of contents. However, because some things are in slightly different places, not all of the offset math works.

In the configuration file a y-axis is a single dimensional table that has an associcated x-axis of the same element count that is used as the header. ie:

rpm  :    0 |   600 |  1000 |  <- x axis
mm^3 :  0.1 |   1.1 |   1.7 |  <- y axis

It does very much appear as though the WinOLS map pack I have was generated from a genuine Cummins configuration for the ISV. However, because winols stores a direct reference to the address of the y-axis parameter, that map is tied to the bin it was made for. This means that winols strips away the id number for that parameter. It also means their file is storing that parameter multiple times.

Since I do know the itn number of the Y axis, that means I have the absolute address for where that parameter is stored in the xcal. WinOLS uses a different addressing method, so I have to take their address for the y axis, then subtract the address for the x axis to find the offset. Then I go back to my table of contents and search for an itn that starts at that address. If I find one and the byte count matches, assign the parameter to that itn.

One of the parameters I looked at, the table was supposed to start 84 bytes before the data, and on that bin, that's the right answer. On my xcal however, it's a few hundred bytes away...

I'll get there.

 

[NavyCuda]

I have been so busy with life and school work, it's pained me that it's been so long since the last update.

So shortly after my last update I basically hit a brick wall. I needed the correct calibration to go with the map pack I'm using so that I can get all of the itns lined up correctly. I looked into getting it, but the way nissan names the .xcals is different than how cummins does it, so I couldn't match the filenames that way. I can't afford to buy every xcal for every ecm part number. This stumbling block plagued me.

Finally Rawze thumped it into my head to just use the bin that came with the mappack, it has the itn table of contents in it, all I need to do is use that table of contents. So I did just that. I still have a fair amount more code to go, but now I'm pulling out Y and X axis data successfully. More importantly I'm getting all the itn numbers which will make it a snap to apply to the correct xcal.

To visualise where I am, I'm at the point in the tunnel where there isn't just light at the end of the tunnel, the exit is getting bigger. I have so much competing for my time, and I have so many more hours of coding to go before I can document the engine... back at it for as long as I can!

 

[NavyCuda]

I've learned some interesting things trying to map the last few hundred parameters...

What I thought was the ecm part number is actually the part number of the calibration. It does not appear as though the ecm itself was ever changed. Another, very interesting thing... the binary in the winOLS map pack and the binary that I know works, they are the same part number. The binary in the winols map pack was extracted from a truck and came with all the fault tables, injector calibrations, etc. The binary in WinOLS and the binary that works are also assembled slightly differently.

I'm guessing here, but this must have to do with either instructions or flags in the binary to signal the end of a block.

Think of the ecm like the industrial area of your town. A block of addresses might be where all the lumber is stored, another block is where all the steel is stored, another where the machine shops are. Well the memory of the ecm is like that too, the parameters and program are stored in blocks. When the ecm is flashed, the tool must install the xcal or the binary to the correct addresses or the ecm will not work. In terms of binarys, I don't understand how these blocks are identified. In the oem calibrations, due to the way intel hex files are written, the block addresses are clear as daylight.

In our trucks there are four address blocks. One really big one that contains the vast majority of the parameters and three smaller ones. The last few hundred itns are in the last block. In winols, and its map pack the addresses are applied to the bytes sequentially starting from 0x00 (0). With intel hex, all the bytes are broken down into records. The record contains its starting address, the number of bytes in that record, the type of record it is and the checksum for that record. This means that the with an intelhex file, the physical address that byte is going to occupy in the microcontroller is specified. The calibration starts at 0x80020000 (2147614720), so that offset is easy. Add the starting offset to the address and you have the correct location. However, the block I'm working in has a different starting address, but there is an unknown number of extra bytes included that tells... something.... something... and I haven't got a clue what they are, what they do... thus my guess at their purpose. This means that even with the correct starting address for that block, there is an additional shift of bytes I haven't figured out how to calculate yet.

I'm thinking these last few hundred will be easier to figure out manually than to try and program something to find the offsets.

School goes to full speed tomorrow and I'll be lucky to get an hour or two a week on this project for the next three months. I'll keep you updated!

 

[Bikerndiver]

This is one of the bedt posts on this forum for sure. Im happy to see someone dig into the truck like this. Keep updating as you can!

 

[NavyCuda]

So I've been trying to find an original xcal (23701EZ45C.XCAL) in the few minutes of spare time I've had. Which isn't much sadly. I haven't been able to find the xcal but I have figured out how to convert the one good binary I have into an xcal. I've started on a binary to xcal converter but my school work is taking up too much of my time and I just can't do anything with it. I wanted to work on it this weekend... but yesterday was all house/family stuff... today is catching up with homework then back to school tomorrow.

Not getting time to work on my big blue turbo diesel makes me sad!

 

[rickk33]

Just read through this entire thread and I am very impressed in your perseverance and drive to self teach yourself this stuff. Regardless of how it turns out well done!

 

[NavyCuda]

So many ideas... no time. School is going decently. only another 10 weeks to go!

I'm almost completely sure this will be a web app when it's complete and offer the ability to document the engine, edit all the parameters and produce a valid xcal or bin. Given how all the cummins engines do the same thing... I'll get our trucks working first then I might attempt expanding out into the dodge rams.

 

[NavyCuda]

Supposedly I'm in week five! I've only had time to pop in and say hi. I've been learning javascript and sql, everything for this project is written in c#... I finally had a bit of spare time last night, so I opened up visual studio, looked at my git log... and spent the whole time trying to figure out how I got to where I am in the code... and where I want to go. Was a bit annoying to accomplish nothing for my beautiful big blue Titan last night.

 

[NavyCuda]

I put a couple of serious hours into my ecm editor today. Regrettably it does not wish to play nice with my new macbook which is a serious set back considering my windows machine is dead. After about an hour I had figured enough of what's going on to try running it and that's where everything fell apart. I don't know if it's an issue with the new macbook being arm and my pc and old macbook being x86... it shouldn't be, but yet I can't get it to run.

It's a salvageable situation but another causality of my primary computer packing it in and me having to replace it with a notebook for work. I really wish I had more hours to work on this.

 

[NavyCuda]

@ehidle

My editor will not look like this... mine will be far more simplistic and just show tables. I have slightly modified these tables which did seem to smooth out the turbo a little bit, but I'm not sure how much more I can take it. I also hope after documenting I'll be able to figure out how to tell it to stop demanding so much drive pressure.

Also not only is there less backpressure from the turbo, but some of the intake mass airflow and "pressure" comes from the egr. The turbocharger seems to always hold a fair amount of drive pressure (2:1), but that exhaust gas is not going into the intake. This means the turbo must spin faster and move more mass in order to satisfy the charge air request.

 

[NavyCuda]

Apparently there was an update to visual studio for apple silicon and now my previous editor works again. Trying to put in a couple hours tonight. I've done so much javascript lately that c# is almost confusing but I'm so glad the editor is running again. Now to figure out where I was and how much more it will take to document out engines!

 

[NavyCuda]

Good news... and bad news...

Bad news... as smart as I like to think I am, my method of learning is not compatible with the web design bootcamp. I'm over a week behind and it's looking very likely that i'm going to be dropped from the boot camp. Working under pressure, in teams and according to someone elses plans is not something I'm skilled at... This is going to throw a serious wrench in the works as it means I have no idea what I'm going to do for work. Injury relief runs out soon and then I'll be on my own, with a blown out back and no real marketable skills.

Good news... I gave up on trying to finish my school work that I'm behind on. So instead what I did was work on truck stuff! Since I'm not working on the editor itself and rather trying to get good data to work with... I started the process of converting the good binary that I have into an xcal document. Once I have the xcal, I'm a day, maybe two away from documenting the engine.

I'm getting too tired to continue on, but the last three hours has been productive. I took the block data structure from the calibration, and verified that the byte count was correct. I believe I've done that before, but I couldn't remember, I'm on a different computer, so I thought I'd make sure.

So as I've mentioned before, our engines have four data blocks, each one has the starting address of the block and the length of it. The binary file is one continuous stream of bytes from start to finish. So as I iterate through each byte of the binary file, I also maintain some counters. if the address gets larger than 0xFFFF, I reset it to zero and add 0x10000 to the extended linear address. If the block position equals the length of the block, I step up the blockIndex plus one and reset all the block related values.

I need to do a bit more testing first, to make sure all the math adds up and I'm not making a +1 or -1 position error while going through all these loops. Once I'm satisfied my loops are clean, then I can start generating the intel hex records, which are the foundation of the calibration. The nice thing is, going over my old code, I had preemptively built some methods just for this job! Too bad I'm getting tired... I'm so close, but the brain is shutting down and I have a meeting with the bootcamp manager types tomorrow to find out my fate.

 

[NavyCuda]

Well my fate is still up in the air as far as school is concerned. They're willing to keep me a bit longer, but have to find out if worksafe is okay with that.

In terms of truck stuff. I've started on an xcal viewer and editor. This will be needed to actually edit the tables and things. I've gotten to the point with converting the bin to an xcal that I need to be able to look at it in an editor of some kind to make sure it's correct. I couldn't find a half decent one for macos with my quick looks... I need more experience in my web design stuff... so I'm just building the editor/viewer in react.

Storybook is a testing tool for testing components of a react application. when I took the screen shot I had the mouse over the one red FF, it also highlights the equivalent plaintext (not correct, just mock up data). I also have it setup so that it will give you the title of each element you mouse over, for the data it will give the absolute address according to the intel hex, for that byte.

I don't know how much more time I can put on this over the next little while, but this is a massive step in the right direction. I need this tool to continue documenting, but this tool will evolve into something that tunes our trucks.

 

[NavyCuda]

I sure hope I don't go over on whatever the allowable bandwidth is for posting!

I'm starting to get excited. All of this is being generated dynamically. I'm using the same test data (the green) for every record, but the important parts work... it's generating an extended linear address record (the short top one), it is assigning the correct address to every single byte! If you hold your mouse over any of the different bytes, data or record details, it either gives you the absolute address of that byte or the notation of what that column represents.

I know, I really know it looks like a big mess of numbers and weirdness, but this is a huge step towards actually having a functioning editor.

 

[NavyCuda]

Today I got the official word that I'm too far behind and I was dumped from the web design bootcamp. It was a lot fun while it lasted, I learned a fair amount, but overall it just wasn't compatible with how I learn. I now enter a highly dynamic period of my life, but that is the nature of these things.

I now have successfully broken up the good bin into not only the correctly shaped block data structure, but I've also further broken it down into the correct extended linear address blocks for the intel hex. Now I have to assemble the data into records and then output an xcal... do some more checks on it... then run it through my other program and hopefully I can finish mapping the remainder of the itns. Once that's done, I can go back into the react app I was building above to verify the data came out correct. I'm going to be spending a lot of time verifying my data... I can't afford to brick my ecm or the engine.

 

[NavyCuda]

So close!

Tiredness is starting to take over so I have to call it a night. However, i'm correctly outputting individual hex records but at the extended address level I seem to be losing a byte or two with each change of address. By the end of the file this adds up to a lot of missing bytes. I'm sure it's a fairly simple bug, but I don't have the mental strength to root it out tonight.

So close, yet just out of reach! Going to be somewhere I got the math wrong... breaking all these bytes up into different components, so many places I could have made a mistake. Nice thing with c# it has a debugger that is pretty straight forward for finding these kinds of issues.

 

[NavyCuda]

Okay, well after bashing my head into my desk I found a number of simple issues with my conversion. The first major issue I had was needing to break the binary into so many separate sections, I had some math issues related to indexes starting from zero, but lengths starting from 1. What this meant is that each segment was losing the last remaining byte due to that index error. The next problem was a logic issue entirely of my own making. Two of the blocks start with an address of zero and an extended linear address of whatever. The last two blocks have an offset that they start at. When I started building the array, I was grabbing positions based off this offset, so instead of grabbing the first byte out of the array, I was grabbing the 700th byte. So, fixed that indexing issue and now I appear to have a clean, properly formatted xcal of the 23710-EZ45C.

Next step, which should, I hope, be easier. Build another converter that converts the xcal back into a pure binary. Then I can compare the original, good binary I have, with the output that has gone back and forth. If they are the same then I can reasonably assume that the conversion is clean on both sides and that it will be safe to go forward with this generated xcal.

Exciting!

Also, it seems I was decent enough that the bootcamp has decided to invite me back, set me into a newer cohort and see if I can make it through this time. So, not next week but the week after I'll be back in school. Hopefully I can get more done this week.

 

[NavyCuda]

While technically it's been "easy" as I know what I need to do, it's turned into quite a few hours of work to go from the xcal to a binary file. This is complicated by the need to keep track of the addresses of every record, extended linear address and the data blocks they reside in according to the block data structure.

├── Binary
│   └── Continuous (Address by simple index, no offsets)
├── Block Data Structure
│   └── Data Block (Starting Address, Length) (target specific area of memory)
│       └── Extended Linear Address Blocks (records assigned to one ELA)
│           └── Records (model of intel hex record)
└── Intel Hex File
    ├── Extended Linear Address (Any following records assigned this value)
    └── Records (Address Specified for this intel hex record)

I've gotten all my code complete to this point, now I just need to test it out. I need to take a break from it for a little while, so thought I'd post another update.