EpochPosition position/pm errors

[mbtaylor]

@lmichel the flattened model is now easier for me to work with, thank you, but I think it may not be quite flattened enough to be consistent.

I see some problems with the errors and correlations between sky position and proper motion components.

Section 5.3.3 (EpochPositionErrors.properMotion) and 5.3.4 (EpochPositionErrors.Position) both say:

Position error: can be an ellipse, a correlation matrix or a covariance matrix"

(aside: probably 5.3.4 should say "Proper motion error" here).

But the off-diagonal elements of these correlations/covariances are already dealt with by the items in Sec 5.2 (EpochPositionCorrelations), namely the items in subsections 5.2.5 (longitudeLatitude) and 5.2.6 (pmLongitudePmLatitude), so if they can be represented under EpochPositionErrors as well it represents an ambiguity.

This point is somewhat related to @mcdittmar's comment on the Gaia example in this thread of the DM list.

Moreover in the gaia_epoch_propagation_flat_full.xml example, the quantities mango:error.ErrorCorrMatrix.sigma1 and mango:error.ErrorCorrMatrix.sigma2 are quoted with units, although correlation matrices would be dimensionless.

What I was expecting for these quantities instead of the two matrix-like items:

EpochPositionErrors.properMotion
EpochPositionErrors.Position

was to see four scalar items:

EpochPositionErrors.longitude
EpochPositionErrors.latitutde
EpochPositionErrors.pmLongitude
EpochPositionErrors.pmLatitude

If you agree, corresponding changes would also need to be made in the example.

If 5.3.3 and 5.3.4 are indeed removed/replaced, does this also remove the need for items in the error package, namely sections 9.3 (ErrorCorrMatrix), 9.4 (ErrorCovMatrix) and maybe 9.2 (Ellipse)?

[mcdittmar]

@lmichel the flattened model is now easier for me to work with, thank you, but I think it may not be quite flattened enough to be consistent.

The flatter it is, the more constrained it gets.
What you're suggesting sounds like it would make EpochPosition even more catered to GAIA data.

I actually think that might be a good thing... a GaiaPosition would be very clear about what context it should be used and help with another concern I noted, about using mango:EpochPosition when meas:Position is all that is needed/wanted.

[lmichel]

In my understanding the correlation matrix relates to the parameters themselves, not the errors.
RAeff = RA + RApmRAcorpmRA + RApmDEcorpmDE

• You are right to say that errors cannot be a covariance matrix, it requires units.
• I do not agree with the idea of flattening more the error since errors on individual quantities can have different shapes which must be specified by a class:

Let's take an example:

• Error on 1D quantities can be symmetrical (defined with one value: +/-err) or asymmetrical (defined with 2 values -e_low +e_high). As the number of parameters required to describe one error changes following the nature of that error, we have to encapsulate it in an object.
• The same argument applies to 2D errors that can be a matrix or an ellipse

I propose:

• remove references to covariance for the EP errors
• rename ErrorCorrMatrix as ErrorMatrix
• remove the ErrrorCovMatrix from the model

[mbtaylor]

OK, I see that there needs to be some way in general to model symmetric and asymmetric 1-d errors. The problem here is with 2-d errors.

I don't understand this comment:

In my understanding the correlation matrix relates to the parameters themselves, not the errors.
RAeff = RA + RApmRAcor_pmRA + RApmDEcor_pmDE

The correlation is the scaled off-diagonal elements of the covariance matrix, and the errors are the square root of diagonal elements of the covariance matrix. So e.g. if a client is looking for the correlation between lat and lon using the current draft, it could look in either

   mango:EpochPositionCorrelations.longitudeLatitude

or

   mango:EpochPositionErrors.position -> ErrorCorrMatrix.Correlation

This redundancy is not helpful when trying to make sense of the annotation.

Although I understand conceptually that sky position and proper motion both seem like 2-d quantities that should have a 2-d covariance/correlation matrix, this is not how the Gaia observational data appears. Because of the way that the Gaia processing is done, the Gaia observations relevant for epoch propagation are:

• a 5-element vector (ra,dec,parallax,pmra,pmdec)
• an associated 5-element error array
• the 10 off-diagonal elements of a 5*5 symmetric correlation matrix
• radialVelocity and its error, which don't have correlations.

(errors and correlations could alternatively be represented using a 5*5 symmetric covariance matrix, though that's not how they appear in the gaia_source catalogue).

Trying to represent this data using 2-d correlation/covariance matrices won't work well (redundancy, missing data or confusing representations will result), which is why I asked for the correlations in the earlier draft to be flattened. A partial flattening doesn't help much.

As per @mcdittmar's remark above, I understand that this is highly specific to Gaia data, but if you want to represent all the statistical information from gaia_source in a straightforward way, it's hard to avoid doing something like this. If you're more concerned with generality than with representation of Gaia data, then you might want to do something else, but I'd still advise to try to avoid redundancy (which is also potential for contradiction).

[lmichel]

To me, the right path is to stay close to the GAIA data (vector + correlations + errors) without closing the door to some flexibility (e.g. error polymorphism)

• I removed the redundancy in Error 2D limited to Symmetrical2D or Ellipse #76 (see the draft diagram)
• I updated the error package in a way that en 2D error can be either an Ellipse or a Symmetrical 2D error with one field per dimension. This latest matches the GAIA pattern.

a 5-element vector (ra,dec,parallax,pmra,pmdec)

Handled by the EpochPosition component.

an associated 5-element error array

Handled by the EpochPositionErrors component.
This one cannot be flater due to the error polymorphism I mentioned above but it does not embed any correlations anymore, it only defines one error value per axis.

the 10 off-diagonal elements of a 5*5 symmetric correlation matrix

Handled by the very flat EpochPositionCorrelations component.

Remark: using matrices in the annotations is very painful for all stack holders since the role of a given element is given by its rank in a matrix row. (Ellipse as been redefined for the same reason)

radialVelocity and its error, which don't have correlations.

Treated as the 6th coordinate of the EpochPosition(Errors)

Notes:

• As the PDF of the source branch is not accessible, it is difficult for the contributors to get a clear picture of the proposed model evolution. I suggest to refer to the diagrams that are located in https://github.com/lmichel/MANGO/blob//model/ (I continue to work with Modelio so that the diagrams are always in sync with the VO-DML).
• I'll update the VOTable once we agree on the class structure and the PR is merged

[mbtaylor]

As long as the errors on positions and proper motions are in fact Symmetrical2D for the Gaia case, this should be OK. If one encountered a MANGO instance where e.g. mango:EpochPositionErrors.position was an ellipse and mango:EpochPositionCorrelations.longitudeLatitude was also present there would be redundancy/conflict, since the ellipse angle is another way of representing the correlation. However I presume the idea is that authors avoid that situation.

I don't suggest using the language of matrices in the annotations, I was just using it here to explain the structure of the Gaia data.

I can build the PDF locally on the branch, so I can see what the amended document looks like.

Once the VOTable is updated, I'll revisit my implementation code.

[lmichel]

You are right, once we have decided to split into 3 separate components (vector, corr, error), we delegate to the annotator the responsibility of keeping these components consistent with each other.
If we merge correlations within errors, ellipse support becomes very problematic, is that a problem?
I don't know, I just know that e.g. the future XMM catalogue should come with elliptical errors, but most of the Vizier catalogues have square error boxes.
I'm open to jumping back in that direction if you prefer.

[mbtaylor]

I'd say it's a policy decision about whether this is supposed to be enabling epoch propagation by modelling the Gaia source catalogue, or providing for more general markup of astrometric data. Since epoch propagation in full detail is a somewhat intricate business, I suspect that it will be difficult to come up with a general solution that fits nicely with different missions. But my guess is that a large majority of epoch propagation at this level of detail (errors and correlations) will be Gaia-based or Gaia-like for quite a few years (decades?) to come. Still, it should be feasible to support entry-level epoch propagation (inputs just position, proper motion and epoch difference) for different datasets, and that can work with just a small subset of the existing EpochPosition elements.

If you want to retain some generality I think it would be OK to have the ellipse option, though perhaps there should be a note that if EpochPositionErrors.Position is ellipse, then EpochPositionCorrelations.longitudeLatitude should (must?) not be used, and if EpochPositionErrors.properMotion is ellipse, then EpochPositionCorrelations.pmLongitudePmLatitude should not be used.

A few other points:

• The text for EpochPositionErrors.Position and EpochPositionErrors.properMotion now says "can be an ellipse or matrix". If "matrix" here I refers to Symmetrical2D, it's a bit misleading, since that component only has 2 free elements. Reword?
• The text for Ellipse.angle says "North Polar Cape (NCP)" - should that be NPC? But I don't know what that Cape is, maybe just North pole would be clearer.
• Requiring the position angle in Ellipse.angle to be positive seems like an unnecessary restriction; there are certainly catalogues out there that have position angles in the range -90..90 rather than 0..180 (see e.g. column w1pa in main.wise at http://dc.g-vo.org/tap).

[lmichel]

Thank you, I'll

• keep my Ellipse
• Write a note in the EochPosistion head about not using both ellipse and correlation
• Clarify the ellipse angle definiton

Next push

[lmichel]

Done in #76 (may need further iterations)