Page Comparison

...

1. CWM Buffering

2. OKE Inbound File Transfer Throttling

3. Aborting an OKE Inbound File Transfer

4. Transferring Large Files

5. Faster Baud Rate Negotiation

6. Fastest Interface

7. File Transfer Throughput

8. File Transfer Status

9. File Status

10. Firmware

11. Writing from the OKC to an OKE System

12. Throttling File Fragments

13. Pseudocode Example

14. Testing File Transfer from Open Kitchen

15. Common Problems

16. Debug Log Output

The CWL allows OKE systems to read and write files containing arbitrary data via the file transfer interface. Individual file transfers of up to 2^31-1 bytes (2GB) in size are allowed. The primary use of file transfer is for firmware updates since it is highly likely that all OKE controllers will support in-system updates. Other possible uses for file transfer include sending recipes to equipment, or for equipment to offload diagnostic logs to the OKC.

...

The CWL makes no assumptions about the filesystem on the implementor's platform, or even if a traditional file system exists. For example, many deeply embedded platforms simply have interfaces that read and write to a non-volatile memory that is partitioned statically. Other platforms, such as Linux, will have full desktop compatible, wear leveling flash file systems. The CWL can easily work with either setup. Implementors should consider the file names used in OKMs to be logical and not necessarily have or demand a direct manifestation as a "real" file.

Anchor
cwm_buffering cwm_buffering

CWM Buffering

File transfers may need to move large quantities of data. Many embedded systems will have limited amounts of high-speed RAM buffers, and there can be significant latency and throughput bottlenecks when writing data out to a non-volatile memory. So how can a small embedded system handle the onslaught of file data sent to it from the cloud computers with 10 Gigabit network connections?

...

By decoupling file transfer between an OKE system from the OKC, the CWM allows an OKE system to easily manage having large files written to it and write large files to the OKC.

Anchor
oke_inbound_ft_throttling oke_inbound_ft_throttling

OKE Inbound File Transfer Throttling

As an OKE receives inbound file write fragments it may easily throttle the CWM from sending the next fragment for up to 4 seconds without triggering serial communication layer timeouts. Implementors should add code to the CWPortIsAppBusy() function that will return true when the application is not ready to handle the next file fragment, and false otherwise. The CWPortIsAppBusy() function must not return true for more than 4 consecutive seconds otherwise serial communication timeouts may occur. Serial communication is always gracefully recovered after a timeout but nevertheless should be avoided.

...

Should the non-volatile memory need to be erased before a write is initiated then implementors should erase the smallest partition necessary to complete the next write to minimize the erase time. For example, it is better to make 16 4KB erase requests that each take 250ms than a single 64KB erase request that takes 4 seconds.

Anchor
aborting aborting

Aborting an OKE Inbound File Transfer

An OKE may abort an inbound file transfer at any time by returning false when CWPortExeFileWrite() is invoked by the library. For example, an OKE may abort or prevent a file transfer if the user enters a technician diagnostic mode. Generally, implementors should minimize or eliminate modes during which file transfers are not permitted.

Anchor
large_files large_files

Transferring Large Files

Files larger than the CWM ISFQ can be transferred from the OKC to an OKE. This is possible by the OKC actively monitoring the progress of the file transfer to ensure that it does not overrun the CWM ISFQ. Using this technique files up to 2GB can be transferred.

Anchor
faster_baud faster_baud

Faster Baud Rate Negotiation

When CW_PORT_DYNAMIC_BAUD_ENABLED is set to 1 in cwport.h the CWL will automatically attempt to negotiate the baud rate defined by CW_PORT_BAUD_RATE_FAST with the CWM. The OKE starts at default baud rate defined by CW_PORT_BAUD_RATE_DEFAULT and the library will try to increase the rate to the desired speed. If the CWM supports running at the higher speed, then both the OKE and CWM will switch. If any communication failures occur or the baud rates get out of sync both the OKE and CWM will drop back to the default baud rate and resynchronize. The OKE will request the baud rate to increase again. Typically, this option will only be used by OKEs using the TTL or 232 bus and that regularly transfer large files >10MB. Other OKEs can simply ignore this option and run at the default baud rate. The allowed values for faster baud rate are 230400, 250000, 500000, and 1000000.

Anchor
fastest_interface fastest_interface

Fastest Interface

The USB interface can support the highest transfer speeds. Although the 232 and USB transfers rates are similar with the current WiFi Wi-Fi CWM hardware, in the future newer CWM hardware will allow the USB interface to go much faster than 1000000 baud.

Anchor
throughput throughput

File Transfer Throughput

The amount of time it takes to transfer a file is a function of many variables including file size, processing latencies, and link bandwidths. The following table shows the estimated throughput between the MiWi and WiFi Wi-Fi CWM versus the OKE connection type:

...

OKE implementors should use these throughput estimates along with their product requirements to help determine the CWM communication port type (USB, serial, etc.) and settings (baud rate, full/half duplex, etc.) and the recommended CWM network type (WiFiWi-Fi, MiWi, etc.) for their application. Poor wireless connectivity or OKE file transfer throttling will reduce throughput so OKE implementors should verify that file transfers are meeting their requirements during the development and integration phase.

...

1. A MiWi CWM connected to an OKE via any communication interface can transfer a 1MB file in about 1 hour.

For the WiFi Wi-Fi CWM the OKE connection to the CWM will be the primary bottleneck for file transfers. Generally, the WiFi Wi-Fi CWM should be able to transfer files to the OKE at a significant fraction of the OKE connection's baud rate.

WiFi Wi-Fi CWM transfer time examples:

1. A WiFi Wi-Fi CWM connected to an OKE via USB can transfer a 1MB file in about 46 seconds.

2. A WiFi Wi-Fi CWM connected to an OKE via TTL full duplex serial at 115200 can transfer a 1MB file in about 3 minutes.

3. A WiFi Wi-Fi CWM connected to an OKE via 232 full duplex serial at 1000000 can transfer a 1MB file in about 50 seconds.

4. A MiWi CWM connected to an OKE via TTL full duplex serial at 115200 can transfer a 1MB file in about 1 hour.

5. At comparable baud rates a half-duplex serial connection will be slightly slower than a full duplex connection for OKC to OKE file transfers.

...

2. A file version to differentiate between different file encodings. This ensures files with incompatible encodings are safely ignored by the OKE.

Anchor
file_transfer_status file_transfer_status

File Transfer Status

When the OKC is writing a file, an OKE system will report on the status of the write operation. This allows the OKC to determine if the file transfer is ongoing, stalled, or completed. If the file transfer appears stalled, for example only half the file was downloaded due to a network connectivity problem. Then the OKC can restart the file transfer from where it left off without starting from the beginning. This can be very helpful when attempting to transfer large files in challenging network conditions. The CWL automatically sends File Transfer Status updates to the OKC.

Anchor
file_status file_status

File Status

Independent of reporting the current status of a file being written to the OKE, the names of important individual files maintained by the OKE are also reported automatically by the CWL.

...

Some examples of files that might make sense to report are the name of recipe configurations, or the name of diagnostic logs. It is not necessary to list the names of firmware files since they are enumerated in the version data. The idea is to enable the equipment to be effectively managed by the OKC. Implementors should think about the capabilities of the product and the use cases for its utilization. Based on this knowledge, provide file information to the OKC that can enhance it manageability.

See https://sitesage.atlassian.net/wiki/spaces/PDOC/pages/493748237/Messages#File-and-Firmware-Transfers for Message status fields to look for.

Anchor
firmware firmware

Firmware

Many systems that implement file transfer will also implement in-system firmware updates. The OKC will transfer firmware images to an OKE system via a file write. An OKE system will be able to write the new firmware image to secondary storage. If the file transfer is successfully and the firmware passes platform specific integrity checks then it should be automatically installed.

Anchor
writing_okc_to_oke writing_okc_to_oke

Writing from the OKC to an OKE System

bool CWPortGetInboundFileWriteBuf(uint8_t** buf, uint16_t* bufSize);

bool CWPortExeFileWrite(int8_t* fileStr, uint32_t offset, uint8_t* data, uint16_t dataLen, int32_t fileSize) ;

...

The CWPortExeFileWrite()  must return true if it successfully processed the file fragment. If an unexpected error occurs, then the function must return false.

Anchor
throttling throttling

Throttling File Fragments

As previously discussed, the CWM buffers all file fragments from the OKC. The OKE can determine the pace at which the file fragments are received in case its local non-volatile memory cannot be written to fast enough. This section describes two strategies for the OKE to implement throttling.

...

Implementors may also consider lazy blocking. For example, after writing data to a non-volatile memory the memory usually goes into a busy state for a period during which it cannot accept new writes. Instead of writing and immediately blocking waiting for the write to complete the CWPortExeFileWrite() function should return without blocking. The next time the CWPortExeFileWrite()  function is called it can block conditionally only if the write operation is still in progress. Deferring the memory busy check can increase file throughput considerably.

...

Pseudocode Example

This example code is from the Windows port of the CWL.

...

Bool CWPortGetInboundFileWriteBuf (uint8_t** buf, uint16_t *bufSize)
{

    static uint8_t ftBuf[CWMSGCMD_FT_MAX_SIZE];

    CW_REQUIRE(buf != NULL);

    CW_REQUIRE(bufSize != NULL);

    *buf = ftBuf;

    *bufSize = sizeof(ftBuf);

    CW_ENSURE(*buf != NULL);

    CW_ENSURE(*bufSize >= CWMSGCMD_FT_MAX_SIZE);

    return true;

}

bool CWPortExeFileWrite (int8_t* fileStr, uint32_t offset, uint8_t* data, uint16_t dataLen)
{
  CW  CW_REQUIRE(fileStr != NULL);
    CW_REQUIRE(data != NULL);

    CW_REQUIRE(dataLen <= CWMSGCMD_FT_MAX_SIZE);

    if (offset == 0)
    {

        if (fpWrite != NULL) fclose(fpWrite); /* Close open files */

        CW_ASSERT((fpWrite = _fsopen(fileStr, "wb+", _SH_DENYNO)) != NULL);

    }

    CW_ASSERT(fwrite(data, 1, dataLen, fpWrite) == dataLen);

    if ((dataLen < CWMSGCMD_FT_MAX_SIZE) && (fpWrite != NULL))
       { fclose(fpWrite);}

    return true;

}

Anchor

...

Testing-File-Transfer-from-Open-Kitchen Testing-File-Transfer-from-Open-Kitchen

Testing File Transfer from Open Kitchen

See the following document and video for instructions on testing file transfer.

https://sitesage.atlassian.net/wiki/x/FwBfFQ

https://www.loom.com/share/85f56abaf2874fa79c844c5e0580eb36?sid=aea09ba1-0528-40cf-890f-d6536f5d3f73

Anchor
common_problems common_problems

Common Problems

1. OKE implementation expects a hardcoded file name and file uploaded to OKC doesn’t match:
   It’s commonly seen that developers hard code an expected file name for certain file types, e.g. firmware.bin, menu.json. If someone subsequently uploads a file to the OKC with a different name (e.g. firmwareV1-2-3.bin), the file will be downloaded to the OKE (and indeed the status shown in Open Kitchen will be ‘downloaded’), but it isn’t loaded into the system because of the name mismatch. We recommend avoiding this approach and to instead encode identifiable information in the file header. If not avoidable, then ensure the file uploaded to the OKC is correctly named

2. Old CWM firmware and/or CWL version: In CWM1 firmware earlier than PIC154/SPH145 and ConnectWare Library versions < 1.0.15, there was a bug in the fragmentation layer that caused messages of certain lengths to be dropped. This can cause a multitude of issues including download rewinds due to dropped fragments (ultimately ended in an aborted download), and files that do ‘successfully' download to the CWM but aren’t loaded into the OKE. If you encounter such issues, please ensure your CWM firmware and CWL versions are up to date.

3. File name + extension longer than 16 characters: the OKC now prevents this from happening, but it may still be possible to do with the simulators. The CWL has a 16 character limit on the file name including the extension. If this limit is exceeded, the CWM will download the fragments from the OKC but just throw them away. Please ensure file names are <= 16 characters including extension.

4. File Name + Version + Account ID not unique combination: Currently, there is a unique index in Open Kitchen Database that requires a combination of File Name - Version - Account ID to be unique even the uploaded file has been deleted. If Version does not have value entered by user, it will have timestamp value. However, user can enter Version for it to be used to match with equipment data for file transfer status. If this is the case, it may cause violation of unique File Name + Version + Account ID combo key. In this scenario, if a firmware file needs to be uploaded from the same Account, it needs to have unique File Name and Version (if input by user) every time it is uploaded even after it is deleted from Open Kitchen.

5. Connectware Module V2 uses low baud rate: Some likely scenarios are the following.

   1. If a CWM2 reports the baud rate as 115200 in its heartbeat, it could be because OKE does not request a baud rate change to increase the RS232 baud rate from 115200 to 230400. CWM2 will accept the baud rate change request from the OKE over the 232 port. The OKE could put some temporary debug output in the CWL where it make the baud rate change requests and processed the baud rate change responses.

   2. If the OKE was connected to a CWM1, and a CWM2 was swapped in without the OKE restarting, then the OKE will not request a baud rate speed increase. This is because the CWM1 cannot run faster than 115200 and informs the OKE that it does not support a speed increase. The OKE will no longer request a speed increase until the OKE’s CWL is restarted. Low baud rate on CWM2 can cause slow file transfer. The baud rate speed is independent of doing a file transfer. Therefore there is no file transfers increased until the CWM reports its baud rate at 230400, since it will never go faster than about 1 hour.

   3. Does OKE run a build that does not change baud rate?

   4. There are some hardware or cabling problem that causes the speed rate change to fail. For example, the speed increase takes effect, but the OKE and CWM cannot communicate successfully at that speed, and eventually revert back to 115200. It is expected that the CWM would sometimes report “baud”:230400 if this was the case.

Anchor
debug_log debug_log

Debugging Log Output

1. The file contents are transferred 512 bytes at a time from the CWM to the OKE, each 512 bytes is considered a fragment. Each one of the “Received… fragment” lines in the log output is showing that the fragment was received where offset is the absolute location of that fragment within the file being transferred. As you can see the offset advances 512 bytes between each log output.

2. Frame errors are usually indicative of hardware signaling problems, not usually software problems. This would more commonly occur at baud rates above 115200. Causes could be poor cabling, poor grounding, or exceeding the maximum bit rate specification of an RS232 transceiver (either side). Some transceivers rated for a maximum of 115200 can sort of work at higher baud rates, but can glitch out and cause frame errors, etc.

3. The serial protocol between the CWM and OKE has a robust ability to detect and recover from temporary errors on the serial communication bus. If you look at the log output and see “Received Fragment” messages after a burst of Frame Error messages, this means the next fragment in the sequence was successfully delivered despite the frame error burst.