0000049: No observations are possible with more than 2048 channels per IVC band
All Projects | CNI | DZB | equalizer | IVC/ADC | LFFE | MFFE | Puma | Puma 2 | RFI | RT | submission | system time | Tadu | Time/LO | TMS | VLBI

Main | My View | View Issues | Change Log | Docs
  

Viewing Issue Simple Details [ Jump to Notes ] [ Issue History ] [ Print ]
ID Category Severity Reproducibility Date Submitted Last Update
0000049 [TMS] major always 05-08-08 12:13 11-11-23 05:09
Reporter Arno Schoenmakers View Status public  
Assigned To Hans van Someren Greve
Priority high Resolution reopened  
Status feedback  
Summary 0000049: No observations are possible with more than 2048 channels per IVC band
Description When doing an observation with more than 2048 channels per IVC band, ProcessDZB cannot process a single 10-sec sample within 10 seconds. This triggers ReadDZB to kill and restart ProcessDZB again and again. As a result, a multitude of empty MSes will be produced.

Based on deep tracing of the ProcessDZB output, the cause seems to be a very inefficient FFT transform in class DZBProcFFT for more than 2048 channels. For less channels, the transform for a whole sample is done within a second. For more channels, it can take several seconds.

Hans van Someren-Greve has been informed, He has adapted the FFT routines a while ago and that must have triggered these problems as these did not exist in the past.
Additional Information
Attached Files

- Relationships

- Notes
(0000052)
Yuan Tang
07-08-08 11:18

 Concerns : New version for ProcessDZB.

I made a change in ProcessDZB (only routine DZBProcFFT).
Only in case the correlation function length N (N is always a power of
2) is equal to 8192 I do a FFT of 8192, in all other cases I do a FFT
on N-1.
The version is called ProcessDZB.8192 and is in production, from
07-08-2008 11:30 local time, starting with 10803701.

Added in the log file of ProcessDZB is 1 line per observation
(TRACE(2)) with the text :
'Execute Real to Complex FFT for nnnn lags'
nnnn can be : 127, 255, 511, 1023, 2047, 4095, 8192, or 16383.
 
(0000053)
Arno Schoenmakers
07-08-08 18:51

 Hallo Arno,

Betreft : FFT voor 4096

Ik neem aan dat je bedoelde een FFT op 8191 correlatie punten, die dan
een spectrum van 4096 punten geeft ?
Als ik nl 4095 correlatiepunten transformeer dan krijg ik een spectrum
van 2048 punten, en dat gaat heel snel.

Mijn testen met het algemene FFT test programma voor TMS, (testfft)
laat ook dat exploderen in tijd zien.

Met dat programma doe ik dan 256 transformen (simulatie van XX en YY
met 1 IVC band) op NPT punten (aantal correlatie punten) de tijden die
ik dan meet zijn :

NPT Tijd
4095 0.144 sec
8191 90.93 sec !!!!!
8192 0.152 sec

De oorzaak van het exploderen in tijd is dat 8191 een priemgetal is.
Het geheim van de FFT zit hem in de factorizering van NPT.
Die is maximaal voor een macht van 2.
4095 gaat ook snel, want dat is : 3x3x5x7x13.

Aangezien 8192 ook weer heel snel gaat, lijkt me de oplossing toch
echt dat we in de ProcessDZB code voor NPT waarden > 4096 op machten
van 2 gaan zitten, en daaronder op machten van 2 min 1.

De penalty is dan alleen een iets grotere ripple, maar dat valt weer
heel goed weg te krijgen met een Taper.

Groeten
Hans
 
(0000054)
Arno Schoenmakers
07-08-08 18:51

 Hi observers, Arno,

Concerns : FFT with 4095 frequency points

I just did with success an observation with 8192 lags (4095
freq. points), the observation is 10803685.

Yuan did an observation with 16384 lags (8191 freq. points), but he
did this with the production version of ProcessDZB.

Because i was somewhat surprised by the explosion in time, (an FFT on
8192 points is more than 600 times faster than a FFT on 8191 points !)
i did some more research.

8191 is a Mersenne number, i.e. it is equal to a power of 2 minus 1,
and it is a prime number.
If you write it as M=2**n-1, then for some time people thought that M
was always a prime if n was a prime.
That is true for M=127 (n=7), and M=8191 (n=13), but not for M=2047
(n=11) because 2047=23x89.

The 'achilleshiel' of FFT's are lengths which are a prime number.
Below are a few examples of lengths, measured durations and uses,
done with the program testfft.
It is immediately obvious that FFT's on prime numbers take much more
time that FFT's on numbers which can be factorized.

NPT Type Time of 10 FFTs (sec) Gebruik door
             forward and backward ProcessDZB
16384 2**14 0.0153 -
16383 3x43x127 0.081 *
16381 prime 17.46
8192 2**13 0.008 *
8191 prime 4.87 niet meer
7193 prime 3.80
6197 prime 3.07
5197 prime 1.91
4093 prime 1.46
4096 2**12 0.0037 -
4095 3x3x5x7x13 0.0084 *
3079 prime 0.62
2053 prime 0.30
2048 2**11 0.002 -
2047 23x89 0.0074 *
1024 2**10 0.0007 -
1023 3x11x31 0.0021 *
127 prime 0.0018 *

The Production ProcessDZB uses only lengths of 2**n-1 and not 2**n
because correlation functions of 2**n-1 are symmetric, and give less
ripple in the spectrum after the FFT (in case there is no Taper).
Only if the length is 8191 there is a problem, even a length of 16383
did not give a problem as Yuan proved, because it can be factorized.

I made a change in ProcessDZB (only routine DZBProcFFT).
Only in case the correlation function length N (N is always a power of
2) is equal to 8192 I do a FFT of 8192, in all other cases I do a FFT
on N-1.
The version is called ProcessDZB.8192 and is not yet in production.
I have to do some more tests.

Question to Arno.
Why do the present log files of ProcessDZB have so much trace 3 and
trace 4 ?
It is very difficult now to find what you want to see.

Groeten
Hans
 
(0000428)
Guest
11-11-23 05:09

 x
 

- Issue History
Date Modified Username Field Change
05-08-08 12:13 Arno Schoenmakers New Issue
05-08-08 12:14 Arno Schoenmakers Assigned To  => Hans van Someren Greve
05-08-08 12:14 Arno Schoenmakers Status new => assigned
07-08-08 11:18 Yuan Tang Note Added: 0000052
07-08-08 11:18 Yuan Tang Status assigned => resolved
07-08-08 11:18 Yuan Tang Resolution open => fixed
07-08-08 18:51 Arno Schoenmakers Status resolved => feedback
07-08-08 18:51 Arno Schoenmakers Resolution fixed => reopened
07-08-08 18:51 Arno Schoenmakers Note Added: 0000053
07-08-08 18:51 Arno Schoenmakers Note Added: 0000054
07-08-08 18:52 Arno Schoenmakers Status feedback => closed
07-08-08 18:52 Arno Schoenmakers Resolution reopened => fixed
11-11-23 05:09 Guest Status closed => feedback
11-11-23 05:09 Guest Resolution fixed => reopened
11-11-23 05:09 Guest Note Added: 0000428