Csv-Parser Reading FFat file instead SD card

[thornhilldenis]

hello dear developer.
i need to Parse a txt and or CSV comma-separated file, I have been searching for an example code of this CSV-Parser to read file from FILESYSTEM FFat on my esp32 for the last 3 days with no success .

Is CSV-Parser only for Sd ?
any reference for using the library from FFAt will be much appreciated.

Thanks, in advance for your time .

Ing Denis Thornhill

[michalmonday]

Hello, the library is mostly independent of underlying file storage. As long as you can read the file into a string (as a whole, or in parts), then you can supply it to CSV_Parser (either in constructor as a whole, or by using << operator and supply it in parts).

I never worked with FFat using esp32 but the following article looks straightforward:
https://www.mischianti.org/2021/04/06/esp32-integrated-ffat-fat-exfat-filesystem-6/

You could combine it with existing examples of CSV_Parser, so the return of the readString() method from image above would either go into constructor of CSV_Parser, or it would be inputted like:
csv_parser_object << testFile.readString();

These examples could be helpful:
basic usage
supplying csv by incomplete parts

[thornhilldenis]

Hello dear developer.
first of all thanks for your response trying to help me out.
I have to spend many days trying to figure out this situation, I have read and read, search and search and tried 100 different combinations and I can not reach my goal.

I have implemented a A.I neural cell on the esp32 wroom using AVR 8.1.5 and Arduino IDE 1.8.19.
My goal is to Load the weights of my A.I Scenario.

USING THE SD CARD IT WORKED LIKE A CHARM (NO PROBLEM AT ALL)
BUT as my Denis.csv file containing my comma-separated weights is so small I really need it on FILESYSTEM INSTEAD as in future will automatically update FILESYSTEM file Denis.csv from an internet Remote server,

SO to go straight to the point :
When using SD card , working Perfectly
CSV_Parser cp(/*format*/ "ffffffffffffffffffffffff", /*has_header*/ true, /*delimiter*/ ','); cp.readSDfile("Denis.csv"); float *WW1 = (float*)cp["WW1"]; float *WW2 = (float*)cp["WW2"];

Now I want to use the same CSV-Parser to parse the same file, BUT NOW from my FILESYSTEM FFat where Denis.csv is uploaded and perfectly READ by the next code
`void parse_ai_weights() {
File readcsv = FFat.open(F("/Denis.csv"), "r");
if (readcsv){
Serial.println("Read file content!");
/// Serial.println(testFile.readString());

String ai_weights = readcsv.readString();
Serial.println(ai_weights);   ///prints on Serial Monitor   PERFECTLY 
readcsv.close();

}
}`

SO I HAVE A String ai_wheights Containing all the data I need to parse.
All the rest I can handle, cause I use the CSV-Parser library almost daily.

BUT I have not been able to correctly code CSV-Parser to read My String ai_wheights to be parsed.

I Tried your proposal as like this But does not compile
CSV_Parser cp(/*format*/ "ffffffffffffffffffffffff", /*has_header*/ true, /*delimiter*/ ','); cp << readcsv.readString(); ////FAILED cp << ai_weights; ////FAILED float *WW1 = (float*)cp["WW1"]; float *WW2 = (float*)cp["WW2"];

So I am kind of desperate Now, if you could help me out I will much appreciate

Denis.csv

please let me know as it looks you have more knowledge than I do.
Many thanks in advance for your time and Kindness

Ing Denis Thornhill

[michalmonday]

Hello, here's a small example. First it creates a file using "writeFile". Then it reads it character by character and supplies it to the CSV_Parser object using cp << file.read();. Then it uses cp.print() to view all parsed values. Then it prints values from VV1 column only.

#include "FS.h"
#include "FFat.h"

#include <CSV_Parser.h>

void writeFile(fs::FS &fs, const char * path, const char * message){
    Serial.printf("Writing file: %s\r\n", path);

    File file = fs.open(path, FILE_WRITE);
    if(!file){
        Serial.println("- failed to open file for writing");
        return;
    }
    if(file.print(message)){
        Serial.println("- file written");
    } else {
        Serial.println("- write failed");
    }
    file.close();
}

const char *denis_csv = "WW1,WW2,WW3,WW4,WW5,WW6,WW7,WW8,VV1,VV2,VV3,VV4,VV5,VV6,VV7,VV8,UU1,UU2,UU3,UU4,UU5,UU6,UU7,UU8\n"
                        "1.529006002,0.253733553,0,0,0,0,0,0,1.284589032,-0.16966153,0.061992747,-0.446341058,-0.453974486,0.293160251,-0.174254381,1.332099744,6.326600076,1.477596343,-1.496865036,9.003172761,-1.129388801,-2.241994092,-0.106482536,0.333588426\n"
                        "-0.217656669,0.496886743,0,0,0,0,0,0,-2.521074192,-0.248332009,2.304620566,3.063565941,-0.084831135,-1.245491029,1.169079723,-2.772691784,15.6392119,-6.559343484,-0.76593996,16.95741907,-0.229103923,-2.174205059,-2.783387003,-4.153871339\n"
                        "-0.685878196,-1.69774026,0,0,0,0,0,0,0.75732003,-0.191953244,2.284465675,0.504071024,-0.457960533,-0.689415175,-0.432202133,-0.505708842,5.091135436,0.857673313,2.886192935,1.381541316,0.413169009,2.899088995,-0.156824798,0.450908032"
                        "-1.56897012,-0.582719571,0,0,0,0,0,0,0.756744231,0.020471757,-0.605422857,-0.580923655,-0.292100131,0.654582639,-0.195243403,1.491637782,0,0,0,0,0,0,0,0\n"
                        "-0.600424066,0.294622049,0,0,0,0,0,0,0.156895215,-0.376825887,0.834245199,0.470031922,-0.25945908,-0.135387552,-0.316315596,0.130135589,0,0,0,0,0,0,0,0\n"
                        "0.75991028,0.84727152,0,0,0,0,0,0,0.021394153,-0.539207383,3.405282902,0.530846276,0.258361585,-0.799486165,0.28407691,-1.038734802,0,0,0,0,0,0,0,0\n"
                        "-1.221449941,-0.028690932,0,0,0,0,0,0,-2.046529053,0.334116574,0.79527923,1.594805338,0.808990216,-0.38662376,1.547053763,-1.93526403,0,0,0,0,0,0,0,0\n"
                        "1.524834584,0.646203694,0,0,0,0,0,0,-1.670471979,0.298746015,1.407371203,2.497918847,0.234797764,-0.900750431,1.002674102,-2.564860142,0,0,0,0,0,0,0,0";

void setup(){
  Serial.begin(115200);
  if(!FFat.begin()){
      Serial.println("FFat Mount Failed");
      return;
  }
  
  writeFile(FFat, "/Denis.csv", denis_csv);

  CSV_Parser cp(/*format*/ "ffffffffffffffffffffffff");
  
  File file = FFat.open("/Denis.csv");
  if(!file || file.isDirectory()){
      Serial.println("- failed to open file for reading");
      return;
  }
  while(file.available()){
      cp << (char)file.read();
  }
  file.close();
  cp.parseLeftover(); // needed after reading the whole file when the last line of the file doesn't end with "\n" 
  cp.print(); // prints all parsed contents

  Serial.println("\nVV1 column:");
  float *VV1 = (float*)cp["VV1"];
  for(int i = 0; i < cp.getRowsCount(); i++) {
    Serial.println(VV1[i]);
  }
}

void loop(){

}

It outputs:

CSV_Parser content:
rows_count = 7, cols_count = 24
   Header:
      WW1 | WW2 | WW3 | WW4 | WW5 | WW6 | WW7 | WW8 | VV1 | VV2 | VV3 | VV4 | VV5 | VV6 | VV7 | VV8 | UU1 | UU2 | UU3 | UU4 | UU5 | UU6 | UU7 | UU8
   Types:
      float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float | float
   Values:
      1.53 | 0.25 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.28 | -0.17 | 0.06 | -0.45 | -0.45 | 0.29 | -0.17 | 1.33 | 6.33 | 1.48 | -1.50 | 9.00 | -1.13 | -2.24 | -0.11 | 0.33
      -0.22 | 0.50 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -2.52 | -0.25 | 2.30 | 3.06 | -0.08 | -1.25 | 1.17 | -2.77 | 15.64 | -6.56 | -0.77 | 16.96 | -0.23 | -2.17 | -2.78 | -4.15
      -0.69 | -1.70 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.76 | -0.19 | 2.28 | 0.50 | -0.46 | -0.69 | -0.43 | -0.51 | 5.09 | 0.86 | 2.89 | 1.38 | 0.41 | 2.90 | -0.16 | 0.45
      -0.58 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.76 | 0.02 | -0.61 | -0.58 | -0.29 | 0.65 | -0.20 | 1.49 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -0.60
      0.29 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.16 | -0.38 | 0.83 | 0.47 | -0.26 | -0.14 | -0.32 | 0.13 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.76
      0.85 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.02 | -0.54 | 3.41 | 0.53 | 0.26 | -0.80 | 0.28 | -1.04 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -1.22
      -0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | -2.05 | 0.33 | 0.80 | 1.59 | 0.81 | -0.39 | 1.55 | -1.94 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 1.52
Memory occupied by values themselves = 768
sizeof(CSV_Parser) = 40

VV1 column:
1.28
-2.52
0.76
0.02
-0.38
-0.54
0.33

The "writeFile" function I used is copied from Arduino IDE -> File -> Examples -> FFat -> FFat_Test which btw contains some more useful functions.

[thornhilldenis]

hi Mr michalmonday
I had had some busy days lately
I am so sorry for the delay in responding to you and being thankful for your response time and explanation.

What I did to fix the problem was
I read my Denis.csv file, as usual, using FFAT
Then passed it to a string
Then passed the String to a CHAR called buf using tochararray, and then CSV_PArser did his job perfectly.

It worked perfectly,
probably is not the correct way of memory use, but later I cleared the variable . and keep going .

Many thanks again for your time and cooperation.
Any comment will be perfect, especially based on my lack of knowledge.

Thanks again, if you agree I will close the issue.

Denis Thornhill

[michalmonday]

Hello, just wanted to let you know that since version 1.0.0 (released today), it is necessary to cast the return of file.read() built-in method to (char) before supplying it to CSV_Parser using << operator, like:

while(file.available()){
    cp << (char)file.read(); // (char) is needed because file.read() returns integer
}

So I edited the example from my previous post. I added a note in README.md explaining why:

Before the 1.0.0 version, the cp << 97; expression would append letter 'a' (because '97' stands for 'a' in ascii table). From 1.0.0 version onwards, the cp << 97; is equivalent to cp << String(97);, it will append '97' instead of 'a'. That is correct behaviour in my opinion, however due to design of Arduino built-in "File.read()" method, which returns an integer, it is necessary to cast it's return (with (char)csv_file.read() as shown above), and problems may occur if some existing code (using this library) doesn't explicitly cast it.