Sample Value calculation and Vgain

The manual say the scope has input voltage ratnges +/- 10, 5, 2.5, 1, 0.5, 0.25, 0.1, 0.05 But the ones calculated in the docs are slightly different.

So i'm wondering. Can the voltage range be configured arbitrarily? in finer steps.
What is the formula for calculating the signal voltage from the signed 10bit sample value?

There is a diagram in the docs that loosely depicts the how samples are amped/attenuated but its kind of hard to make sense of it when the names in the hw config doc don't match the code. I assume bit 0 of the "ets_on; adc_interleaving; Analog_sw" field is also K1.

The docs also say that for vgainX 0xFF25  is 0 dB what is it for other hex values?
How does offsetX factor in?

Thanks in advance.

The input voltage ranges +/- 10, 5, 2.5, 1, 0.5, 0.25, 0.1, 0.05 are full scale ranges. In the docs (AnalogInputScaling.ods), the range is specified as volts per division. You can convert voltage to full scale range (2 V per division is equal to +/- 10 V full scale range).

So i'm wondering. Can the voltage range be configured arbitrarily? in finer steps.
What is the formula for calculating the signal voltage from the signed 10bit sample value?

It could be possible to adjust the voltage range arbitrarily, but this is currently not implemented in the software. The input signal goes through the variable gain amplifier (VGA AD8337) which gain can be adjusted from the control DAC MAX5501 (the software is setting the gain values in the VgainA and VgainB registers). The VGA gain adjustment is linear in decibels, the formula for VGA voltage gain can be found in the AD8337 datasheet. The calculation for gain is also used in the AnalogInputScaling spreadsheet, but note however that these calculations are only an approximation. The actual gain will vary because of the offset and gain variations of individual hardware components.
The calibration process that is implemented in the software is adjusting the gain more accurately. During the calibration the gain is determined iteratively (not calculated from a formula). The procedure will set the AWG generator to a fixed voltage (for example 1 V) and then the offset and gain is adjusted iteratively to match that level on the grid. For example, if gain is set to 1 volt per div., the signal should be at first division ->  ADC 10-bit value will be 1024/10 divisions = 102. At the end of the calibration procedure the gain that was required to set that voltage will be saved in the calibration settings.

Regarding the control register bits:
- Analog_sw register contains attenuation bits (bit1: att-A, bit0: att-B): These bits will select the input attenuation K1 (the RC network attenuation). The  att-A bit is switching the input relay for channel 1
- adc_interleaving will switch the RE7 relay - this will connect the analog input channel 1 to both ADC inputs (and disconnect the analog input channel 2)
- ets_on is used to select the ETS sampling mode

Ah ok. I was confused because the doc said 0xFF25  is 0 dB. After looking in the datasheet and schematic i see that 0xFF25  is approximately 12db. or simply since -0.6v is 0db and +0.6v is 24db, 0db is approximately 0x555 and 24db 0xD55.

Anyhow,
I can easily calibrate the offset for zero by shorting the inputs, but the gain is more complicated since the calibration value is dependent on the accuracy of the reference voltage.
I do not have a voltage reference with which to calibrate the ADC or a voltmeter to calibrate the DAC so i need another way.
Are there calibration values from manufacturing i can read out and use to calibrate the DAC settings?
What is the approximate gain of the DAC?

Attached is the offset required to achieve 0v when the input is shorted vs the vgain for both channels. does that make sense?

The values you have measured for offset are looking correct.

You can use AWG output for gain calibration (select DC - adjust offset). The same method is used for factory calibration.
Values from factory calibration are stored in EEPROM. They can be read using the software menu: EPROM -> Read Calibration (software must be run "as administrator"). After selecting this command, the calibration data will be read out from EEPROM to the file on disk.

The calibration data will be located
in windows:    <scopefun_installation_dir>/data/startup/hardware2.json
on linux:    /usr/lib/oscilloscope/data/startup/hardware2.json

Below is an example of offset calibration data:

   "callibratedNormal":   {
      "offsetsCh0":   {
         "volt2":   -47,
         "volt1":   -48,
         "mili500":   -51,
         "mili200":   -60,
         "mili100":   -47,
         "mili50":   -48,
         "mili20":   -52,
         "mili10":   -61
      },
      "offsetsCh1":   {
         "volt2":   -23,
         "volt1":   -25,
         "mili500":   -27,
         "mili200":   -35,
         "mili100":   -23,
         "mili50":   -25,
         "mili20":   -28,
         "mili10":   -35
      },

And similar for gain calibration data:

      "gainValueCh0":   {
         "volt2":   1423,
         "volt1":   1777,
         "mili500":   2129,
         "mili200":   2596,
         "mili100":   1417,
         "mili50":   1768,
         "mili20":   2244,
         "mili10":   2599
      },
      "gainValueCh1":   {
         "volt2":   1425,
         "volt1":   1779,
         "mili500":   2129,
         "mili200":   2587,
         "mili100":   1419,
         "mili50":   1770,
         "mili20":   2241,
         "mili10":   2594
      },